jason810496 opened a new pull request, #71704:
URL: https://github.com/apache/airflow/pull/71704
- closes: #69001
- supersedes: #69774, #69007
- obsoletes: #71276
## Why
The scheduler cached deserialized Dags in a dict that never evicted, so
stale Dag versions were never cleared and the process was eventually OOM
killed. A TTL cache with no size limit fixes it: versions are reclaimed once
their Dag runs finish and they stop being requested.
`[api] dag_cache_size = 0` had a related gap. It selected that same
never-evicting dict and silently ignored `[api] dag_cache_ttl`, so TTL eviction
could not be enabled without also accepting a size limit.
## How
- `DBDagBag.__init__` picks the mapping from both options instead of gating
on `cache_size` alone: a TTL gives a `TTLCache` sized by `cache_size` or
uncapped, a size alone gives an `LRUCache`, neither gives the plain dict.
Negatives clamp to 0 with a warning.
- Introduce `APIServerDBDagBag` and `SchedulerDBDagBag`. Each subclass only
sets its metric prefix and resolves its own config section; the cache logic and
the `_stat_*` hooks stay on the base, so both components emit the same set of
metric suffixes.
- Add `[scheduler] dag_cache_size` (default 0, no size limit) and
`[scheduler] dag_cache_ttl` (default 3600).
- Teach the metrics registry check to match dynamic names by regex over
their static parts, which is what lets the hooks live on the base. This
replaces #71276: matching only the prefix before the first variable cannot
resolve a name whose variable comes first, and it also cannot handle a variable
in the middle such as `{prefix}.foo.{state}.duration`. The regex handles both.
### TTL semantics
`_get_dag` re-assigns the key when it revalidates an entry, and cachetools
resets expiry on `__setitem__`, so **revalidating an entry resets its expiry**.
The TTL therefore reclaims only the Dag versions that stop being requested for
the whole interval; versions still referenced by active Dag runs are kept. That
is what reclaims the accumulation this PR is about, but it means
`dag_cache_size` stays the only hard ceiling.
Separately, neither option governs staleness, which the existing `[api]
dag_cache_ttl` docs implied. A Dag update that creates a new version is picked
up immediately, since the new version is a different cache key. A version
rewritten in place is re-checked against its current `dag_hash` once `[core]
min_serialized_dag_update_interval` has elapsed, so that option bounds how long
a rewritten version can be served stale. The config reference and FAQ now say
this for both components.
### Scheduler lock impact
Caching flips the scheduler's `_dags` from a plain dict guarded by
`nullcontext` to a cachetools mapping guarded by an `RLock`, so
`get_dag_for_run` now takes a lock it did not take before.
`dev/airflow_perf/dag_bag_cache_overhead.py` replays `_get_dag`'s hit path to
separate the two costs:
| configuration | ns/lookup | vs today |
|---|---|---|
| `dict` + `nullcontext` (scheduler today) | 147 | baseline |
| `dict` + `RLock` | 143 | -3 |
| `LRUCache` + `RLock` | 281 | +134 |
| `TTLCache` uncapped + `RLock` (this PR) | 560 | +414 |
| `TTLCache` uncapped + `nullcontext` | 568 | +421 |
The lock is not a cost at all: an uncontended `RLock` is consistently
*cheaper* than the `nullcontext` it replaces, because
`RLock.__enter__`/`__exit__` are C-level slots while `contextlib.nullcontext`'s
are pure-Python methods paying full frame setup. That is why the `dict` +
`RLock` row comes out negative — it reproduces under reversed ordering and
interleaved A/B runs, so it is a real property of the comparison, not noise.
The whole delta comes from the `TTLCache` bookkeeping itself, so dropping the
lock would buy nothing while giving up a real thread-safety invariant. At 10k
`get_dag_for_run` per scheduling loop that is ~4 ms/loop, dwarfed by the
`DagVersion` lookup `get_dag_for_run` already issues before it ever consults
the cache. The `RLock` stays.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Opus 4.5 following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
