denysivanov commented on issue #18598: URL: https://github.com/apache/airflow/issues/18598#issuecomment-934834141
@potiuk Currently I have in PROD 186 dags. I would have around 500 by new year + 500 - 1000 dags during 2022. We set up Grafana in prod and I see that DAG processing total parse time is 4.8 minutes. I looked up this article https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html and tried to play with it in QA. I created a copy of 1000 dags from that sample and did some tests in my QA You can see details here [DAG load process.docx](https://github.com/apache/airflow/files/7289250/DAG.load.process.docx) Surprisingly those details are very different from the case when I am running it locally. And moreover they are not what article describes ... Here is how my environment look like in QA & PROD  In case of QA & PROD I am getting my variables from HashiCorp Vault. For DEV it is local. I do not expect that it should be matter but it is... So currently I am not traveling to HashiCorp vault and take parameters from local environment + I did min_file_process_interval = 300 dag_dir_list_interval = 300 scheduler_parsing_processes = 4 and this addressed my issue (I do not produce dynamic dags so after deploy I am not touching them at all) This change brought DAG processing total parse time from 4.8 minutes to 3.3 seconds. Let me know if you have to add anything here from your perspective :) Thanks a lot for your help. I do really appreciate it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
