akhilpratap1991 opened a new issue, #71272:
URL: https://github.com/apache/airflow/issues/71272
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.2
### What happened and how to reproduce it?
### What happened
The scheduler enters an unrecoverable crash loop. On startup and on every
`[scheduler] orphaned_tasks_check_interval`,
`adopt_or_reset_orphaned_tasks()`
builds a log message by calling `repr(ti)` on each orphaned `TaskInstance`.
`TaskInstance.__repr__` reads `self.state`, which is a deferred column. When
the
`TaskInstance` is detached from its SQLAlchemy session, that read triggers a
deferred DB load with no bound session and raises `DetachedInstanceError`.
The
exception is unhandled in `_run_scheduler_loop`, so the scheduler process
exits.
Because the scheduler dies *before* it finishes resetting the orphaned task,
the
same orphan is still present on the next start and it crashes again — a
self-perpetuating loop. Nothing outside the scheduler clears the orphan, so
it
never recovers without manual DB intervention. While it loops, no DAGs are
scheduled at all.
Traceback (paths are from the official `apache/airflow` image):
```
[error ] Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File ".../airflow/jobs/scheduler_job_runner.py", line 1546, in _execute
ret = execute_callable()
File ".../airflow/jobs/scheduler_job_runner.py", line 1612, in
_run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File ".../airflow/utils/session.py", line 100, in wrapper
return func(*args, session=session, **kwargs)
File ".../airflow/jobs/scheduler_job_runner.py", line 2814, in
adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
...
File ".../airflow/jobs/scheduler_job_runner.py", line 2864, in
adopt_or_reset_orphaned_tasks
reset_tis_message.append(repr(ti))
File ".../airflow/models/taskinstance.py", line 1134, in __repr__
return prefix + f"[{self.state}] ti_id={self.id}>"
File ".../sqlalchemy/orm/attributes.py", line 569, in __get__
return self.impl.get(state, dict_)
...
File ".../sqlalchemy/orm/strategies.py", line 536, in _load_for_state
raise orm_exc.DetachedInstanceError(
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <TaskInstance at
0x...>
is not bound to a Session; deferred load operation of attribute 'state'
cannot
proceed (Background on this error at: https://sqlalche.me/e/20/bhk3)
### How to reproduce
Using KubernetesExecutor:
1. Start a task so a `TaskInstance` is `running`/`queued` under a worker pod.
2. Cause it to become "orphaned" — e.g. the worker pod is lost, or the
scheduler is killed while the task is in flight (SIGKILL / OOM / node
drain)
so the owning job is gone.
3. Start the scheduler. `adopt_or_reset_orphaned_tasks()` selects the
orphaned
`TaskInstance`(s); when it reaches `reset_tis_message.append(repr(ti))`
the
deferred `state` load fails with `DetachedInstanceError` and the scheduler
exits. It then crash-loops on every restart because the orphan persists.
Manual workaround: set the stuck task instances to a terminal state directly
in
the metadata DB (or via the UI "Mark Failed"/CLI from a non-scheduler pod),
then
restart the scheduler.
### What you think should happen instead?
Adopting/resetting orphaned tasks must not be able to crash the scheduler,
and
in particular building a diagnostic log line must never trigger a lazy DB
load
on a possibly-detached instance. `adopt_or_reset_orphaned_tasks()` should
either:
- construct the message from the task instance identity/key (already-loaded
columns) instead of `repr(ti)`, or
- make `TaskInstance.__repr__` read only already-loaded attributes (e.g. from
`__dict__`) so it degrades gracefully (`state=?`) rather than raising, or
- keep the instances bound/refreshed within the session while the message is
built.
The task instances should still be reset as intended; only the fatal log call
needs to change.
### Operating System
Debian (official `apache/airflow:slim-3.2.2-python3.12` image), Python 3.12.
### Deployment
Other
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes (KubernetesExecutor). Reproducible
with
the version pinned by the 3.2.2 constraints file.
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
Happens repeatedly — every scheduler restart while an orphaned task instance
exists. Related (same code path / theme, but not the same `__repr__`/`state`
lazy-load crash): #30709, #39088, #19671, #58570, #13808.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
