GitHub user pedro-cf edited a comment on the discussion: DAG Parsing Performance with Large Number of Dynamically Generated DAGs
@potiuk My issue is that I have 1 dag file dynamic_dags.py that queries the database and based on the configuration it creates specific dags. When the number of dag configs exceeds 2000 the parsing takes to long. How can I split this workload through multiple dag processors without manually creating multiple dag files with a query skip/limit? GitHub link: https://github.com/apache/airflow/discussions/44727#discussioncomment-11664195 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
