ChickenBenny opened a new pull request, #76191: URL: https://github.com/apache/airflow/pull/76191
closes: #76190 --- ## What In a Dag with `fail_fast=True` and an unversioned bundle (such as the default local Dags folder), removing or renaming a task while a run is active makes the next final task failure in that run raise `KeyError` in `_stop_remaining_tasks`. The run resolves to the latest Dag version, which no longer has the task, but the run still has a TI for it. This causes three problems: - **Execution API:** the worker's `PATCH .../state` (FAILED) returns 500. The catch-all in `ti_update_state` applies fail_fast again and hits the same error, so the TI is never marked FAILED. - **Scheduler:** the executor reports the task as failed, the scheduler calls `ti.handle_failure()`, and the same `KeyError` makes the scheduler process exit. - **Restart loop:** on restart the TI, still `running`, is reset as orphaned and runs again. `retries` is bypassed and the loop never ends. In a local repro, a task with `retries=0` reached try 13 in about 30 minutes. ## Fix `_stop_remaining_tasks` now fetches only unfinished TIs and skips TIs whose task is no longer in the Dag. Both the Execution API path and the scheduler paths go through this function. The filter matches what marking a Dag run as failed already does in `_set_dag_run_terminal_state` (`airflow/api/common/mark_tasks.py`). Not-yet-scheduled TIs (`state IS NULL`) are still skipped, because `State.unfinished` includes `None`. Two behaviour notes for reviewers: - Finished TIs other than `SUCCESS`/`FAILED` (`UPSTREAM_FAILED`, `REMOVED`, `SKIPPED`) are no longer rewritten to `SKIPPED`. - A still-running TI of a removed task is not stopped by fail_fast itself. Once the scheduler marks it `REMOVED`, its next heartbeat gets a 409 and the supervisor terminates it. Out of scope, possibly for follow-ups: the scheduler does not isolate an exception from `handle_failure` for a single TI, and the `ti_update_state` catch-all re-runs fail_fast handling. The same code is in 3.3.x, so this may be worth backporting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
