viiccwen opened a new pull request, #69815:
URL: https://github.com/apache/airflow/pull/69815
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
closes #69814
Repeated API requests that resolve the latest Dag always need to query the
latest serialized row to preserve freshness. However, they also deserialized
that row on every request, even when the same Dag version and hash were already
cached.
This change reuses the cached deserialized Dag when the queried version ID
and hash are unchanged.
It keeps the existing latest-row query, reloads serialized Dags that were
updated in place, and refreshes the cache validation timestamp. The cache
documentation now distinguishes freshness revalidation from LRU/TTL eviction.
A real-path benchmark measured the ORM latest-row query, Dag
deserialization, and task route handler. Each result is the median of three
rounds with 50 lookups per round; lower is better.
| Backend | Serialized payload | Before | After | Speedup |
| --- | ---: | ---: | ---: | ---: |
| SQLite | 31 KB | 3.052 ms | 0.437 ms | 6.98x |
| SQLite | 153 KB | 13.487 ms | 0.965 ms | 13.98x |
| SQLite | 305 KB | 28.765 ms | 1.808 ms | 15.91x |
| PostgreSQL 14 | 31 KB | 3.798 ms | 1.002 ms | 3.79x |
| PostgreSQL 14 | 153 KB | 16.461 ms | 2.093 ms | 7.87x |
| PostgreSQL 14 | 305 KB | 30.390 ms | 3.912 ms | 7.77x |
| MySQL 8 | 31 KB | 4.139 ms | 1.033 ms | 4.01x |
| MySQL 8 | 153 KB | 15.952 ms | 1.933 ms | 8.25x |
| MySQL 8 | 305 KB | 29.914 ms | 3.371 ms | 8.87x |
The benchmark uses a new SQLAlchemy session for each lookup and leaves
`[core] compress_serialized_dags` at its default value (`False`). It excludes
HTTP middleware, authentication, and response encoding, so the table isolates
the route-handler path affected by this change rather than claiming the same
percentage improvement for total HTTP latency.
Tests cover matching cache hits, validation timestamp refresh, cache
metrics, cache misses, and in-place hash changes. The DagBag test file, related
API/query-count tests, Ruff checks, pre-commit checks, manual checks, and
Apache Airflow documentation build pass.
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X] Yes — Codex (GPT-5.6 Sol)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
