potiuk commented on issue #59445: URL: https://github.com/apache/airflow/issues/59445#issuecomment-3714539655
One of the reasons might also be not upgrading to latest version. 3.1.5 had some "N+1" dag fixes (too many sql queries generated on dag list page) . What can also slow down things are huge history in dag files that might slow down queries (`airflow db clean` might help). Maybe those are limited resources on your database, not vacuumed/cleaned/rebuilt indexes - the usual database maintenance that should happen regardless of what application is using it. You should take a loook at the monitoring of your database, looking at excessive queries, peaks in resource usage, strange errors - possibly looking at plans of queries generated by your database and looking for full-table scans. Generally when database statistics are not updated and cleaned up regularly, your table stats migh wildy differ from the actual data stored and database engine might use sub-optimal ways of planning the queries, whihc - for example -might result in loading all rows to memory instead of just loading indexes. Also we are just about (next week latest) to have RC of 3.1.6 which you might want to take for a spin before we release it (watch the devlist for announcement) and it has even more small fixes and optimisations in those UI/DB queries generated and it might be that this will help. Another thing you might take a look at - Fast API (when run in containers) recommends ONE worker per container and multiple containers - so comparing to flask it's way better to have several running api-servers - each in a separate container rather than several workers in a single api-server (as counter-intuitive it might seem). You can try it all - ideally one-by-one and see if things are improving after every step - it would be great to have feedback if some of that helped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
