GitHub user pykenny edited a comment on the discussion: Need help understanding total number of Dags oscillating on UI
Sorry, I seemed to made the wrong statement about the behavior earlier -- DAG processor updates the metadata database, and scheduler tries to infer updated state from database as well, they're not making information exchange about the DAGs directly. When running standalone parser, scheduler look up the timestamp and stale duration to decide if some DAGs are out-of-sync, and deactivate them as needed. So perhaps there are some job overlapping between processor and scheduler, but I'm not certain about the assumption/idea behind the design (for instance, should scheduler deactivate all the DAGs eventually when DAG processor's down?). GitHub link: https://github.com/apache/airflow/discussions/44495#discussioncomment-12235149 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
