When running Airflow with a single dynamic DAG file generating multiple DAGs Airflow the scheduler no longer forks processes on a per DAG basis and instead only forks on a per DAG file basis. We are trying to understand why this decision was made and if there are any suggestions on how to improve scheduler performance when an airflow instance contains one or more DAG files that generate many DAGs.
-Andy Cooper
