steveahnahn opened a new pull request, #72669: URL: https://github.com/apache/airflow/pull/72669
On default (unversioned) bundles, redeploying a Dag without one of its tasks while a run still has that task unfinished makes the scheduler erase the pending work and complete the run as success. The success callback fires although a task failed and was waiting to retry. Reproduced with the scheduler's real version resolution and `update_state`: ``` GATE before: b=up_for_retry, run=running, bundle_version=None GATE scheduler resolves run to task set: ['a'] GATE after: b=removed, run=success ``` ### Root cause A run with no bundle version always resolves to the latest Dag version. `task_instance_scheduling_decisions` then hits `TaskNotFound` for the vanished task and sets the instance to REMOVED unconditionally, clobbering UP_FOR_RETRY, QUEUED, RUNNING, or DEFERRED. REMOVED instances are excluded from run-state evaluation, so the run completes green. This behavior dates to 2.0.1, which fixed a scheduler crash by choosing unconditional removal. The same file already disagrees with it: `verify_integrity` deliberately refuses to remove instances of a running run. ### Fix An unfinished instance whose task no longer exists is now failed instead of removed, and counts as an effective leaf (it has no downstream in the Dag), so the run finishes failed and the log says why. Finished instances keep the existing REMOVED behavior. The same scenario now yields: ``` GATE after: b=failed, run=failed ``` The failure callback fires instead of the success one. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5) Generated-by: Claude Code (Fable 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
