The GitHub Actions job "Tests (AMD)" on airflow.git/1fanwang/aip97-failure-details has failed. Run started by GitHub user 1fanwang (triggered by 1fanwang).
Head commit for run: e87f0ba671e466aa6b8e0725ab89565769a27496 / 1fanwang <[email protected]> AIP-97: failure context propagation and transparent infra retry Today a task-failure listener sees only the worker-side exception, and every failure spends a user retry — an eviction looks the same as a real bug, so infrastructure disruptions silently burn the user's retry budget. This adds a small, opt-in vocabulary for why a task failed: - on_task_instance_failed gains failure_kind: TaskFailureKind — a str enum (INFRA / APPLICATION / TIMEOUT / MANUAL) in airflow_shared.state, next to the existing AssetStateStoreWriterKind. Pluggy dispatches by name, so listeners that don't declare it keep working; being a str enum, listeners that want a string still compare equal to "infra"/"application"/"timeout"/"manual". It is None when the cause was not classified. - The executor's short reason token (Evicted / Preempting / ...) is persisted on the TaskInstance row as infra_reason, mirroring retry_reason (AIP-105) — queryable history, read off task_instance, not a transient payload. - Opt-in refund: when failure_kind is INFRA, [core] infra_failure_refund_retries refunds the attempt (bumps max_tries) instead of charging the user's retries, bounded by [core] max_infra_refunds. Application, timeout, and manual failures never refund. - The Kubernetes executor classifies a failed pod (classify_pod_failure): a node-level disruption (Evicted / Preempting / NodeShutdown / DisruptionTarget), or a pod removed while its task was running (PodDeleted — drain, preemption, spot reclaim, force-delete), is INFRA; a container that ended on its own — an app crash, or an OOM against its own limit — is APPLICATION, so an app OOM earns no refund. The classification rides a side channel on BaseExecutor the scheduler reads once. The refund is a single gate keyed on failure_kind == INFRA: a SIGTERM to a running task is ambiguous (a manual mark-failed and an infra eviction both send one), so the decision keys on the source, never the signal — and INFRA is only assigned when the executor reports the task dead while its TI state is still RUNNING (the state mismatch). Every user-initiated stop (mark task/dagrun failed or succeeded, clear) writes the terminal state first, so it is never misclassified. The scheduler stays conservative: an unclassified death refunds nothing — only a positive infra read from an executor bridge does. Behavior is unchanged unless [core] infra_failure_refund_retries is enabled; the failure_kind on the listener is additive and always available. Signed-off-by: 1fanwang <[email protected]> Report URL: https://github.com/apache/airflow/actions/runs/30121240550 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
