GitHub user lakabosch added a comment to the discussion: DAG runs perfectly when I trigger it manually, but when it runs on its scheduled interval, some tasks fail with AirflowTaskTimeout errors
This is most likely caused by scheduler overload, the scheduler can't keep up with parsing and scheduling all DAGs. What you can do is to: - Increase scheduler resources - Reduce parsing frequency ``` # Increase scheduler resources [scheduler] max_threads = 4 # Increase from default 2 parsing_processes = 2 # If you have multiple CPU cores scheduler_heartbeat_sec = 3 # Faster heartbeat detection # Reduce parsing frequency [core] dagbag_import_timeout = 30 # Increase timeout min_serialized_dag_fetch_sec = 15 # Reduce frequency for serialized DAGs ``` GitHub link: https://github.com/apache/airflow/discussions/58363#discussioncomment-14980557 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
