viiccwen opened a new issue, #69814:
URL: https://github.com/apache/airflow/issues/69814
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
`main` (`3.4.0`)
### What happened and how to reproduce it?
The API server's `DBDagBag.get_latest_version_of_dag` queries the latest
serialized row and
deserializes its payload on every lookup. This happens even when the
configured LRU/TTL cache
already contains the same Dag version and serialized hash.
The latest-row query is required to preserve freshness, but repeating
`DagSerialization.from_dict()` for unchanged data bypasses the expensive
part of the cache.
Steps to reproduce:
1. Enable the API server Dag cache with a positive `[api] dag_cache_size`.
2. Store a serialized Dag and create a process-lived `DBDagBag`.
3. Request the latest Dag by `dag_id` twice, for example through a route
that calls
`get_latest_version_of_dag`.
4. Observe that both requests load the latest serialized row and deserialize
its payload, even
though the second request resolves to the cached version and hash.
The overhead grows with the serialized payload. In a real-path benchmark
covering the ORM query,
Dag deserialization, and task route handler, an approximately 31 KB payload
took 3.1–4.1 ms per
request across SQLite, PostgreSQL, and MySQL. An approximately 305 KB
payload took 28.8–30.4 ms.
### What you think should happen instead?
The API server should continue querying the latest serialized row, then
reuse the cached
deserialized Dag when both `dag_version_id` and `dag_hash` are unchanged. If
the hash changed for
the same version ID, it should discard the stale cache entry and deserialize
the current row.
This preserves freshness while avoiding repeated deserialization. The same
benchmark measured
0.4–1.0 ms for the approximately 31 KB payload and 1.8–3.9 ms for the
approximately 305 KB payload
with this behavior.
### Operating System
Reproduced in the Apache Airflow Breeze development environment on macOS.
### Deployment
Other Docker-based deployment
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
