1fanwang opened a new pull request, #73350: URL: https://github.com/apache/airflow/pull/73350
Kubernetes can stop a healthy task without an application exception, but Airflow's failure listeners have no shared category for why it failed. Platform integrations must interpret backend-specific text to route the failure. This implements [AIP-97 Task Failure Classification](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=399279312). Workers and executors report an optional failure kind and diagnostic reason to infrastructure listeners. Failure metrics carry bounded category labels, and logs retain the available cause. Kubernetes classification requires documented disruption evidence; an OOM kill, worker loss or bare eviction reason remains unclassified. Ordinary retries, clears and Dag callback context stay unchanged. This adds no configuration, schema change or replacement-attempt accounting. [AIP-116](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=451974654) covers retry policy separately. Related: https://github.com/apache/airflow/pull/66405 is the combined prototype; this PR contains only classification and reporting. ## Testing Done The [complete runtime fixture](https://gist.github.com/ba08bdda0e5bfee365e11f3be8a8851d/76430f757954c822553cac97a30be20865d5a733) is public and includes every script and dependency. It runs real Kubernetes Task SDK workers, scheduler and API processes against PostgreSQL and captures DogStatsD packets. It compares `6edb01fc98a4900328120aaff67771f471e37026` with baseline `257fe6e85112580547374b6e6d6e5b5576155536`. After downloading the fixture, set `AIRFLOW_CHECKOUT` to the Airflow checkout and run: ```bash python3.12 -B run.py --source "$AIRFLOW_CHECKOUT" --commit 6edb01fc98a4900328120aaff67771f471e37026 python3.12 -B verify_evidence.py runs/<directory-printed-by-run.py> ``` The tasks reached `RUNNING` through authenticated API calls and sent later heartbeats before disruption. No task row or Kubernetes condition was fabricated. The same NoExecute taint that had no shared cause on baseline now reaches the listener as `infra`: ```json {"utc": "2026-09-18T17:34:49.562687+00:00", "origin": "scheduler", "dag_id": "aip97_classification_infra", "task_id": "probe", "run_id": "manual__infra_5fd9a227", "try_number": 1, "kind": null, "reason": null} {"utc": "2026-09-18T17:39:33.006741+00:00", "origin": "scheduler", "dag_id": "aip97_classification_infra", "task_id": "probe", "run_id": "manual__infra_e5ed5b65", "try_number": 1, "kind": "infra", "reason": "DeletionByTaintManager"} ``` The state comparison confirms that this does not grant attempts. With `retries=0`, the disrupted task fails on try 1 with `max_tries=0` on both revisions. With `retries=1`, an application failure retries and succeeds on try 2; clearing it then produces try 3. Callback sequences match baseline. | Scenario | Observed classification | Retry behavior | |---|---|---| | NoExecute taint | `infra`, `DeletionByTaintManager` | Failed without an extra attempt | | Task exceeds an emptyDir limit | `None`; metric label `unclassified` | Failed without an extra attempt | | Application exception or execution timeout | `application` or `timeout` | Existing configured retries | | Ordinary retry followed by API clear | `application` on the failed attempt | State and callbacks match baseline | | Public API marks a running task failed | `manual` in the API listener | Existing terminal state; no failure counter on this route | The verifier decoded the raw UDP captures and checked each authenticated start and later heartbeat: ```json { "passed": true, "cases": 12, "attempts": 16, "behavior_equal": true, "total_udp_datagrams": 30211, "candidate_udp_datagrams": 15502, "policy_counter_packets": 0 } ``` <details><summary>Additional raw listener output</summary> ```json {"utc": "2026-09-18T17:40:22.901520+00:00", "origin": "scheduler", "dag_id": "aip97_classification_storage", "task_id": "probe", "run_id": "manual__storage_2d1dc117", "try_number": 1, "kind": null, "reason": "Evicted"} {"utc": "2026-09-18T17:40:36.610254+00:00", "origin": "worker", "dag_id": "aip97_classification_application", "task_id": "probe", "run_id": "manual__application_f77a91c1", "try_number": 1, "kind": "application", "reason": null} {"utc": "2026-09-18T17:41:18.985456+00:00", "origin": "worker", "dag_id": "aip97_classification_timeout", "task_id": "probe", "run_id": "manual__timeout_be0aa4db", "try_number": 1, "kind": "timeout", "reason": null} {"utc": "2026-09-18T17:42:02.024226+00:00", "origin": "api", "dag_id": "aip97_classification_manual", "task_id": "probe", "run_id": "manual__manual_d79e2f81", "try_number": 1, "kind": "manual", "reason": null} ``` </details> This ran on local kind with DogStatsD. The disrupted task exited 137 after its 30-second grace period; cooperative SIGTERM finalization, live AWS and Celery were not exercised. Repository validation gates also completed. - [ ] Local code review completed --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes, GitHub Copilot CLI Generated-by: GitHub Copilot CLI following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
