jedcunningham opened a new pull request, #61273:
URL: https://github.com/apache/airflow/pull/61273
The grid structure endpoint was loading all serdags for the shown dagruns
into memory at once, before merging them together.
Now, we load 5 at a time and also expunge so they can be gc'd more quickly.
For normal simple dags this makes nearly no difference:
Before:
Average: 0.0163 secs
Requests/sec: 184.5453
95%% in 0.0225 secs
After:
Average: 0.0177 secs
Requests/sec: 169.2285
95%% in 0.0214 secs
But for "edge case" dags - those with many serdag versions shown on a single
grid, and large serdag versions, it makes a significant difference and can be
the difference between an healthy api server and one that OOMs.
Before:
<img width="1616" height="297" alt="before_bad"
src="https://github.com/user-attachments/assets/461c307c-2ebc-486e-9cb5-cf8466b5655c";
/>
Average: 12.5568 secs
Requests/sec: 0.0796
90%% in 12.9824 secs
After:
<img width="1621" height="261" alt="after_bad"
src="https://github.com/user-attachments/assets/3bf2ada0-55e2-4b03-97a1-8e609bf80b3c";
/>
Average: 12.2856 secs
Requests/sec: 0.0814
90%% in 12.4549 secs
(This profiling was done with an intentionally bad dag to really highlight
the difference)
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [x] Yes (please specify the tool below)
Generated-by: Cursor (Claude 4.5 Opus)
Generated-by: Gemini CLI (Gemini 3 Pro)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
