GitHub user pykenny edited a comment on the discussion: Need help understanding total number of Dags oscillating on UI
When scheduler and DAG processing runs independently (enable `standaline_dag_processor`), scheduler has no access to DAG definition folders, and need some protocol between DAG processor to know when a DAG definition gets removed from the file system and should be deactivated. Standalone DAG processor tries its best to send the information (discovering new ones/updating existing ones/removal) but still has some limitation, hence `dag_stale_not_seen_duration` is a bottom line to make the final decision when DAG processor stops sending any information about some DAG currently managed by the scheduler. `min_file_process_interval`, on the other hand, is the last line of defense for DAG parser to ensure a DAG definition's status by force parsing definition file constantly, in case it fails to detect change in the file system correctly. GitHub link: https://github.com/apache/airflow/discussions/44495#discussioncomment-12231354 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
