GitHub user pykenny edited a comment on the discussion: Need help understanding total number of Dags oscillating on UI
When scheduler and DAG processing runs independently (enable `standaline_dag_processor`), scheduler has no access to DAG definition folders ~~, and need some protocol between DAG processor to know when a DAG definition gets removed from the file system and should be deactivated~~. Standalone DAG processor tries its best to ~~send the information (discovering new ones/updating existing ones/removal)~~ update DAG Bag for the scheduler but still has some limitation, hence `dag_stale_not_seen_duration` is a bottom line to make final decision when DAG scheduler stops receiving any information about some managed DAG from DAG processor. `min_file_process_interval`, on the other hand, is the last line of defense for DAG parser to ensure a DAG definition's freshness by force parsing definition file constantly, in case it fails to detect change in the file system correctly. GitHub link: https://github.com/apache/airflow/discussions/44495#discussioncomment-12231354 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
