john-jac commented on issue #14583: URL: https://github.com/apache/airflow/issues/14583#issuecomment-823574467
> I'm not sure if there is one (or even range) universal values to be considered healthy that match all. > For example `dag_processing.total_parse_time | Seconds taken to scan and import all DAG files once` > > What is the appropriate value that can be recommended? Consider an Airflow instance with 3 dags and instance with 3000 Dags. What would be helpful is recommended guidelines, regardless of the number of DAGs (as the number of DAGs don't linearly equate to parse time given other variables like DAG complexity). For example, if Total Parse Time > x minutes, how does that effect task execution? Does that mean that a user should add more resources to the scheduler? If so, should they add memory/cpu/both? Another option would be mapping values like "if you have metric x, then you should have a configuration value of x". Going back to the parse time example, if I have a parse time of 2 minutes then I should set processor_poll_interval to X and min_file_process_interval to Y -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
