GitHub user rtrindvg edited a comment on the discussion: Need help understanding total number of Dags oscillating on UI
Found the issue in our case. It was option "c". After some discussion with the data team, which is the one which creates the DAGs, it was clear no best practices were being used. Besides large imports (which did not need to happen during the parse stage), there were some costly operations happening, like connecting to a vault to get secure parameters and also some database connections. This multiplied by the number of affected DAGs clearly explained the observed behavior. Thanks again for the detailed post which made my life a lot easier for them to understand it was not an infrastructure issue. GitHub link: https://github.com/apache/airflow/discussions/44495#discussioncomment-11513531 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
