tirkarthi opened a new issue, #61485: URL: https://github.com/apache/airflow/issues/61485
### Apache Airflow version main (development) ### If "Other Airflow 3 version" selected, which one? _No response_ ### What happened? In Airflow 2 when a dag page is loaded the grid is loaded along with the details tab. Loading a dag page had only 4-5 requests for grid data, details, asset events etc. In Airflow 3 since the dag page now uses an API call per dagrun it results in 10 API calls along with API calls for other parts of the page like structure_data, plugins, config, backfills, hitl, dag warnings, recent dag runs, recent task instances, asset events, latest run resulting in total of 25-30 API calls. Along with the fact that auto refresh is 3 seconds many of these API calls are made again once the initial API call finishes. Each API call has a set of dependencies and we noticed that `requires_access_dag` is checked per request for a ti_summary. For 10 requests to load the grid this results in 10 checks though this never changes in between the API calls. We also noticed that per `requires_access_dag` also fetches the team for the dag which also doesn't change often, We enabled caching these checks with an in-memory cache and this helped in noticeable performance improvements. This also reduces the db load with the requests served faster helping workers to process other requests. This also helps with other endpoints where `require_access_dag` check is performed redundantly. This involves caching the `is_authorize_dag` check which users can do by customizing the auth manager but I opened this up for discussion. Git branch with patches : https://github.com/tirkarthi/airflow/tree/cache-requires-check ``` time hey -c 4 -H 'Cookie: session=07b3594d-405a-489b-ae0c-02ee02c9bbdb.uq376NuwgAarpTdjToOPTJuO2a0; _token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwianRpIjoiZWVhN2RhODBkZjFjNGRmMjkxZWRmMzE1YzBlNTZmNDQiLCJhdWQiOiJhcGFjaGUtYWlyZmxvdyIsIm5iZiI6MTc3MDI3NDU0MCwiZXhwIjoxNzcwMjc4MTQwLCJpYXQiOjE3NzAyNzQ1NDB9.gVjrIncHUIswAGwQ4IHR-nbmfrLWzQFWPSl1M1wex_Vm9De8DOjAI8bM1v5mOGOfXil-Y5jp0bGGVp4rSlwwSw' 'http://localhost:8000/ui/grid/ti_summaries/asset_produces_2/manual__2026-01-31T05:00:20.973352+00:00' Summary: Total: 0.6636 secs Slowest: 0.0171 secs Fastest: 0.0098 secs Average: 0.0132 secs Requests/sec: 301.3933 Response time histogram: 0.010 [1] |■ 0.011 [2] |■ 0.011 [3] |■■ 0.012 [17] |■■■■■■■■■■■■■ 0.013 [46] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.013 [54] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.014 [37] |■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.015 [25] |■■■■■■■■■■■■■■■■■■■ 0.016 [8] |■■■■■■ 0.016 [5] |■■■■ 0.017 [2] |■ Latency distribution: 10% in 0.0119 secs 25% in 0.0124 secs 50% in 0.0131 secs 75% in 0.0139 secs 90% in 0.0146 secs 95% in 0.0154 secs 99% in 0.0170 secs Details (average, fastest, slowest): DNS+dialup: 0.0000 secs, 0.0098 secs, 0.0171 secs DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0004 secs req write: 0.0000 secs, 0.0000 secs, 0.0002 secs resp wait: 0.0130 secs, 0.0095 secs, 0.0168 secs resp read: 0.0002 secs, 0.0000 secs, 0.0008 secs Status code distribution: [200] 200 responses hey -c 4 -H 0.08s user 0.05s system 19% cpu 0.672 total ``` After caching `requires_access_dag`, teams and serialized dag retrieval ``` time hey -c 4 -H 'Cookie: session=07b3594d-405a-489b-ae0c-02ee02c9bbdb.uq376NuwgAarpTdjToOPTJuO2a0; _token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwianRpIjoiZWVhN2RhODBkZjFjNGRmMjkxZWRmMzE1YzBlNTZmNDQiLCJhdWQiOiJhcGFjaGUtYWlyZmxvdyIsIm5iZiI6MTc3MDI3NDU0MCwiZXhwIjoxNzcwMjc4MTQwLCJpYXQiOjE3NzAyNzQ1NDB9.gVjrIncHUIswAGwQ4IHR-nbmfrLWzQFWPSl1M1wex_Vm9De8DOjAI8bM1v5mOGOfXil-Y5jp0bGGVp4rSlwwSw' 'http://localhost:8000/ui/grid/ti_summaries/asset_produces_2/manual__2026-01-31T05:00:20.973352+00:00' Summary: Total: 0.3493 secs Slowest: 0.0283 secs Fastest: 0.0047 secs Average: 0.0069 secs Requests/sec: 572.6096 Response time histogram: 0.005 [1] | 0.007 [168] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.009 [24] |■■■■■■ 0.012 [3] |■ 0.014 [0] | 0.017 [0] | 0.019 [0] | 0.021 [1] | 0.024 [0] | 0.026 [1] | 0.028 [2] | Latency distribution: 10% in 0.0057 secs 25% in 0.0061 secs 50% in 0.0064 secs 75% in 0.0068 secs 90% in 0.0074 secs 95% in 0.0084 secs 99% in 0.0283 secs Details (average, fastest, slowest): DNS+dialup: 0.0000 secs, 0.0047 secs, 0.0283 secs DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0006 secs req write: 0.0002 secs, 0.0000 secs, 0.0205 secs resp wait: 0.0063 secs, 0.0043 secs, 0.0093 secs resp read: 0.0003 secs, 0.0000 secs, 0.0182 secs Status code distribution: [200] 200 responses hey -c 4 -H 0.07s user 0.05s system 31% cpu 0.354 total ``` ### What you think should happen instead? _No response_ ### How to reproduce 1. Load the dag page with 10 dagruns. 2. Add a print statement to `requires_access_dag` function which is printed per API call though access to dag in one instance indicates this check is not required for future ti_summary API calls. ### Operating System Ubuntu 20.04 ### Versions of Apache Airflow Providers _No response_ ### Deployment Virtualenv installation ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
