ChickenBenny opened a new issue, #76530:
URL: https://github.com/apache/airflow/issues/76530
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
main; the same code path is present in 3.3.2
### What happened and how to reproduce it?
Follow-up from #76191 (fixing #76190).
`ti_update_state`
(`airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py`)
has a catch-all `except Exception` whose purpose is to mark the task instance
`FAILED` when an unexpected error happens during the state update. In that
branch it rolls back, then calls `_handle_fail_fast_for_dag()` again:
```python
except Exception as e:
...
session.rollback()
ti = session.scalar(select(TI).where(TI.id ==
task_instance_id).with_for_update(of=TI))
...
query = query.values(state=(updated_state := TaskInstanceState.FAILED))
if ti is not None:
_handle_fail_fast_for_dag(ti=ti, dag_id=dag_id, session=session,
dag_bag=dag_bag)
```
If the original exception came from fail_fast handling, the second call
raises the same error. Nothing catches it, so the request returns 500 and the
task instance is never marked `FAILED`. The fallback meant to guarantee a
terminal state is what prevents it. In #76190 the error was a `KeyError` for a
task removed from the Dag, and the worker's `PATCH .../state` kept failing.
#76191 removes that specific `KeyError`, but any exception raised from
`_handle_fail_fast_for_dag()` still takes this path.
### What you think should happen instead?
The fallback branch should always be able to write `FAILED` for the task
instance. Either it should not re-run fail_fast handling, or it should run it
in a way where an error there does not prevent the state update, for example a
separate try that logs and continues.
Open point: whether fail_fast should still be applied on this fallback path
at all. Skipping it means the remaining tasks of the run are not stopped early
in this rare case, while running it risks repeating the original failure.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
- Related: #76529
- Related: #76191
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
