GitHub user potiuk added a comment to the discussion: Why three schedulers are all sending UPDATE dag SQL to backend meta db?
No. they might do the same "work" - i.e. parse the same files and update their serialized and code. This is built-in in airflow scheduler design - DagFileProcessor is not supposed to "avoid" parsing the same file by multiple processors. There are various ways you can control it: * you can have standalone dag file processor * those processors can work only on a specified folders (but two dag file processors will still be able to parse and update info about the same DAG files) * https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#file-parsing-sort-mode "random_seeded_by_host" sorting order is implemented so that your multiple schedulers (or multiple dag file processors) are not processing the DAG files in the same order - but randomly - to avoid the possibility that your dag file processors constantly process the same files at the same time. That allows you to decrease the parsing frequency as effectively your multiple processors will be processing the same files more frequently - but in different order. GitHub link: https://github.com/apache/airflow/discussions/44141#discussioncomment-11296503 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
