tirkarthi commented on issue #62020: URL: https://github.com/apache/airflow/issues/62020#issuecomment-3909779291
Thanks for the meta issue. I have been also profiling and testing our performance in the internal environment. For learning purposes I have created a repo to just implement the latest dag_run endpoint in Go, Starlette and FastAPI skipping auth and dag authorization checks. I have found that directly using async driver like aiopg/aiomysql to skip SQLAlchemy usage but still use pydantic delivers up to 2x improvement. Go is up to 10x faster with low resource usage which is not surprising . Another issue to look is around performance of FastAPI under no load and high concurrent load. Loading a dag detail page with 10 dagruns makes close to 40 API calls at the same including 1 API call per dagrun in the grid. As opposed to Airflow 2 where only 4 calls were made for the grid object data which was split in Airflow 3 to become an API call per dagrun in the grid. Under initial load of dag page the API time is high and once the UI loading is done as backend is free re-running the same API call from Chrome/Firefox one at a time is much faster. So there needs to be some analysis on the effects of concurrent requests being made from UI on loading a dag page. Repo : https://github.com/tirkarthi/airflow-go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
