GitHub user salimuddin07 added a comment to the discussion: Airflow 3.0.6 Kubernetes Executor Fails on First "ErrImagePull"
This is not a Kubernetes bug — it’s how the Airflow Kubernetes Executor currently interprets pod status events. Kubernetes behavior: When you get an ErrImagePull, Kubernetes itself doesn’t kill the pod permanently. It retries a few times (backoff). Often it will succeed later, and the pod transitions to Running. >From Kubernetes’ point of view, that’s fine — a transient pull error isn’t an >automatic failure. Airflow behavior: The Kubernetes Executor listens to pod status updates. When it sees ErrImagePull (or sometimes ImagePullBackOff), it interprets that as a terminal failure and immediately marks the task as failed. That’s why your TaskInstance goes into failed, and when the pod eventually comes up, Airflow rejects it with invalid_state. So yes — what you’re seeing is current intended behavior in Airflow, but arguably it’s too aggressive for transient registry issues. There’s already been community discussion about whether Airflow should treat ErrImagePull as a retryable event instead of an immediate failure. What you can do now Switch imagePullPolicy to IfNotPresent (as suggested). Since you already version your images with tags/names, you won’t miss updates. This avoids unnecessary pulls and reduces transient ErrImagePull issues. Use Airflow task retries as a safety net. If Airflow still marks tasks as failed on pull error, configure retries at the task level (retries, retry_delay). That way, a temporary registry hiccup won’t kill your DAG run. (Optional) Raise/track upstream issue This behavior has been discussed before, and it may be considered a usability gap in Airflow’s Kubernetes Executor. If you rely on Chainguard, GitLab, or a flaky registry, it’s worth opening or watching the GitHub issue so Airflow could handle ErrImagePull more gracefully in future versions. Right now, Airflow treats ErrImagePull as a hard failure — that’s why tasks get marked failed immediately. It’s not a Kubernetes problem, it’s how the Kubernetes Executor works. For production stability: Set imagePullPolicy: IfNotPresent (since you tag images anyway). Add task retries to absorb transient image pull failures. Optionally track the Airflow issue — long-term, Airflow may improve this behavior. GitHub link: https://github.com/apache/airflow/discussions/56049#discussioncomment-14582485 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
