kaxil opened a new pull request, #67850:
URL: https://github.com/apache/airflow/pull/67850
Follow-up to #67822. Fixes the root cause of #67813: a scheduler
`DetachedInstanceError` crashloop on startup when the KubernetesExecutor is in
use.
`adopt_or_reset_orphaned_tasks` selects orphaned TaskInstances under `FOR
UPDATE SKIP LOCKED` and, in the same transaction, hands them to
`executor.try_adopt_task_instances`. For the KubernetesExecutor that path
reaches `_alive_other_scheduler_job_ids` (added in #66400), which opened a
**scoped** `create_session()`. Executors run in the scheduler thread, so a
scoped session resolves to the scheduler's own in-flight session. The context
manager's `commit()`/`close()` on exit then:
- commits the scheduler's transaction early, releasing the `FOR UPDATE SKIP
LOCKED` row locks, and
- closes the session, detaching the orphaned TaskInstances the scheduler
still holds.
The reset loop then touches those detached instances (`repr(ti)`, then
`prepare_db_for_next_try` -> `record_ti`, then the `state = None` writes) and
raises `DetachedInstanceError`. The triggering rows persist, so every restart
re-crashes (CrashLoopBackOff); with HA, all schedulers die.
#67822 hardened the `repr()` read, but the crash recurs at `record_ti` (it
reads `try_number` and copies every column off the same detached instance), and
the premature commit / lock release was never addressed. This PR fixes the
cause.
## Fix
```python
with create_session(scoped=False) as session:
...
```
`scoped=False` uses `settings.NonScopedSession`, a separate session and
connection, so the helper's commit/close affect only itself. The scheduler
keeps its transaction, its row locks, and its attached TaskInstances, and
everything downstream works unchanged.
## Why not thread the caller's session into the helper
That would re-introduce the coupling this removes, and is worse: a query
error inside the helper would poison the scheduler's locked-row transaction. An
independent read is the correct boundary.
## Tradeoff
`scoped=False` checks out a second pooled connection while the scheduler
holds row locks. This is a cold path (once per scheduler loop, throttled by
`_last_completed_pod_adoption`) and the default pool has headroom
(`pool_size=5` + `max_overflow=10`). On an extreme `pool_size=1,
max_overflow=0` config the checkout could wait up to `pool_timeout`, after
which the pre-existing `except Exception` fallback degrades to exclude-self
adoption (and logs a warning).
Fixes #67813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
