deepujain opened a new pull request, #75726: URL: https://github.com/apache/airflow/pull/75726
**Title:** Stop retries when the retry callback raises AirflowFailException ## Summary A task can use its retry callback to decide that another attempt would be too late or too costly. Raising `AirflowFailException` there currently only logs a callback error, so Airflow retries the task anyway. This change lets the callback stop retries and fail the task. ## Changes Retry callbacks run before the final task outcome is reported. If one raises `AirflowFailException`, remaining retry callbacks stop and normal failure handling runs with the original task exception, including the failure callback, listener and failure email. Ordinary callback errors still allow the retry. A supervisor message starts the existing finalization timeout while callbacks run. It retains the original retry payload as a fallback if the process exits without reporting an outcome; an explicit failed report or server termination takes precedence. This keeps callback execution bounded without reporting the retry before its callback has finished. Prior work: #60415, #64198, #66781 and #73849 explored this bug but closed without merging. #63342 was a closed empty PR for the same issue. Losing the final outcome message remains a separate recovery limitation. ## Finalization cost A local MsgPack socket benchmark alternated five base/head runs of 5,000 iterations with a no-op callback. The extra supervision message adds about 30 microseconds when a retry callback is configured, so the callback can decide the outcome while retaining its timeout. The unchanged JSON parsing control moved by 5.6%; the successful-task result is included as a control, with no speedup claim. | Path | Base median | Proposed median | | --- | --- | --- | | Retry finalization with callback | 52.26 µs | 82.34 µs | | Success finalization | 39.30 µs | 29.48 µs | | JSON parsing control | 20.12 µs | 18.99 µs | ## Test plan - [x] Reproduced the original incorrect retry with a failing regression test. - [x] Passed the full Task SDK suite: 4,220 tests passed and 6 skipped on Linux arm64 with Python 3.12. - [x] Passed a configured Dag-test reproducer: the stopping callback fails after try 1, an ordinary callback error allows try 2, and a direct `AirflowFailException` fails after try 1. - [x] Covered callback process exit without an outcome, explicit failure, server termination, IPC send errors and shared finalization overtime. - [x] Passed scoped pre-commit and manual checks, including type checks and schema regeneration. - [x] Passed both Go suites (`go test ./...` and `go test -race ./...`) and all 935 TypeScript SDK tests. - [x] Passed the TypeScript SDK documentation build and the full Python documentation build. The first broad build ran out of memory; the unfinished packages passed with one build process. - [x] Ran the selected core database and non-DB suites. The combined non-DB run had 5,520 passes, 11 skips, 2 expected failures and 12 executor-loader failures. Those 12 reproduce on unchanged main after a CLI test reloads the mocked class; they pass in isolation on both main and this candidate. - [x] The provider database run had 4,383 passes, 316 skips and 30 SSH/SFTP failures or errors. The non-DB run had 27,496 passes, 271 skips, 1 expected failure and 37 failures. All 67 failed cases passed in isolated recovery with normal Breeze service initialization; the original combined runs remain failed. - [x] Passed the full provider type check with other large suites stopped. Earlier attempts ran out of memory. - [x] On the current head, passed 32 callback lifecycle tests, both protocol compatibility tests, the 12 isolated executor cases and the 3 serialization timeout recovery cases. The full SDK and broad suites above ran before the rebase; Git confirms the issue patch is unchanged. - [ ] Hosted CI and review. closes: #60172 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (OpenAI Codex) Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
