mik-laj commented on a change in pull request #10531:
URL: https://github.com/apache/airflow/pull/10531#discussion_r475983480
##########
File path: docs/scheduler.rst
##########
@@ -18,11 +18,41 @@
Scheduler
==========
-The Airflow scheduler monitors all tasks and DAGs, then triggers the
-task instances once their dependencies are complete. Behind the scenes,
-the scheduler spins up a subprocess, which monitors and stays in sync with all
-DAGs in the specified DAG directory. Once per minute, by default, the scheduler
-collects DAG parsing results and checks whether any active tasks can be
triggered.
+The scheduler is the core component in Airflow that is responsible for
monitoring all tasks and DAGs and triggers the task instances once their
dependencies are complete. And for this reason, it is imperative to learn about
the working of the scheduler.
+
+Scheduling in airflow involves the 3 following components.
+
+1. ``DAGFileProcessor`` - Responsible for parsing the DAG definition in a file
and creating the necessary DAG runs and TaskInstances
+
+2. ``DAGFileProcessorManager`` - Responsible for listing the files in the
DagBag and creating new DAGFileProcessors when required
+
+3. ``SchedulerJob`` - Responsible for sending the TaskInstances to the executor
+
+Logic behind scheduling is as follows:
+
+1. Airflow kicks off DAGFileProcessorManager along with the SchedulerJob. The
DAGFileProcessorManager enumerates all the files in the DAG directory.
+
+2. The DAGFileProcessorManager spawns child processes known as
DAGFileProcessor which is responsible for creating the necessary DAG Runs and
TaskInstances. When it determines that task instances should run, it updates
their state to ``SCHEDULED``. The number of DAGFileProcessor to spawn is
configurable via ``num_runs`` in ``airflow.cfg``. TODO: Check if the
configuration is correct
+
+3. If any of the DAGFileProcessor have finished, another process is spawned to
work on the next file in the series.
+
+4. The cycle is repeated for all the files in the DAG Bag. If the process to
parse DAG file is still running when the file's turn comes up in the next
cycle, the file is skipped and the next file in the series will be assigned for
the new processor. This isolation provides a non-blocking DAG parsing
functionality.
+
+To illustrate better,
+
+.. image:: img/scheduler_flow.png
+
+TODO: Do we add missing task instance states in the TaskInstance document
+
+If the task instance has a state of NONE it will be set to SCHEDULED if the
scheduler determines that it needs to run.
+Tasks in the SCHEDULED state are sent to the executor, at which point it is
put into the QUEUED state until it actually runs. TODO: Add links to
TaskInstance in the documentation. The reason that the task instances are
created in the ``SCHEDULED`` state, but then are set to the ``QUEUED`` state
once it is sent to the executor, is to ensure that a task instance isn't
repeatedly send to the executor if the executor is slow and a DAG definition
file is processed multiple times before the executor has a chance to run the
task. When the DAGFileProcessor examines a DAG for potential tasks to put into
the ``SCHEDULED`` state, it skips those task instances in the ``QUEUED`` state.
+
+Since the scheduler can run indefinitely, it's necessary to periodically
refresh the list of files in the DAG definition directory. The refresh interval
is controlled with the ``dag_dir_list_interval`` configuration parameter. In
cases where there are only a small number of DAG definition files, the loop
could potentially process the DAG definition files many times a minute. To
control the rate of DAG file processing, the ``min_file_process_interval`` can
be set to a higher value. This parameter ensures that a DAG definition file is
not processed more often than once every ``min_file_process_interval`` seconds.
Review comment:
```suggestion
Since the scheduler can run indefinitely, it's necessary to periodically
refresh the list of files in the DAG definition directory. The refresh interval
is controlled with the ``dag_dir_list_interval`` option in ``[scheduler]``
section. In cases where there are only a small number of DAG definition files,
the loop could potentially process the DAG definition files many times a
minute. To control the rate of DAG file processing, the
``min_file_process_interval`` option in ``[scheduler]`` section can be set to a
higher value. This parameter ensures that a DAG definition file is not
processed more often than once every ``min_file_process_interval`` seconds.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
