potiuk commented on a change in pull request #18356:
URL: https://github.com/apache/airflow/pull/18356#discussion_r712927003
##########
File path: docs/apache-airflow/concepts/scheduler.rst
##########
@@ -180,10 +338,43 @@ The following config settings can be used to control
aspects of the Scheduler HA
SchedulerJobs.
This setting controls how a dead scheduler will be noticed and the tasks it
- was "supervising" get picked up by another scheduler. (The tasks will stay
- running, so there is no harm in not detecting this for a while.)
+ was "supervising" get picked up by another scheduler. The tasks will stay
+ running, so there is no harm in not detecting this for a while.
When a SchedulerJob is detected as "dead" (as determined by
:ref:`config:scheduler__scheduler_health_check_threshold`) any running or
queued tasks that were launched by the dead process will be "adopted" and
monitored by this scheduler instead.
+
+- :ref:`config:scheduler__dag_dir_list_interval`
+ How often (in seconds) to scan the DAGs directory for new files.
+
+- :ref:`config:scheduler__file_parsing_sort_mode`
+ The scheduler will list and sort the DAG files to decide the parsing order.
+
+- :ref:`config:scheduler__max_tis_per_query`
+ The batch size of queries in the scheduling main loop. If this is too high,
SQL query
+ performance may be impacted by one or more of the following:
+
+ - reversion to full table scan - complexity of query predicate
+ - excessive locking
+
+ Additionally, you may hit the maximum allowable query length for your db.
+ Set this to 0 for no limit (not advised).
+
+- :ref:`config:scheduler__min_file_process_interval`
+ Number of seconds after which a DAG file is parsed. The DAG file is parsed
every
+ min_file_process_interval number of seconds. Updates to DAGs are reflected
after
+ this interval. Keeping this number low will increase CPU usage.
+
+- :ref:`config:scheduler__parsing_processes`
+ The scheduler can run multiple processes in parallel to parse DAGs. This
defines
+ how many processes will run.
+
+- :ref:`config:scheduler__processor_poll_interval`
+ The number of seconds to wait between consecutive DAG file processing.
Review comment:
I will explain it (and we might want to rename the parameter separately)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
