potiuk commented on a change in pull request #18356:
URL: https://github.com/apache/airflow/pull/18356#discussion_r712052857
##########
File path: docs/apache-airflow/concepts/scheduler.rst
##########
@@ -187,3 +279,36 @@ The following config settings can be used to control
aspects of the Scheduler HA
:ref:`config:scheduler__scheduler_health_check_threshold`) any running or
queued tasks that were launched by the dead process will be "adopted" and
monitored by this scheduler instead.
+
+- :ref:`config:scheduler__dag_dir_list_interval`
+ How often (in seconds) to scan the DAGs directory for new files.
+
+- :ref:`config:scheduler__file_parsing_sort_mode`
+ The scheduler will list and sort the dag files to decide the parsing order.
+
+- :ref:`config:scheduler__max_tis_per_query`
+ The batch size of queries in the scheduling main loop. If this is too high,
SQL query
+ performance may be impacted by one or more of the following:
+
+ - reversion to full table scan - complexity of query predicate
+ - excessive locking
+
+ Additionally, you may hit the maximum allowable query length for your db.
+ Set this to 0 for no limit (not advised)
+
+- :ref:`config:scheduler__min_file_process_interval`
+ Number of seconds after which a DAG file is parsed. The DAG file is parsed
every
+ min_file_process_interval number of seconds. Updates to DAGs are reflected
after
+ this interval. Keeping this number low will increase CPU usage.
+
+- :ref:`config:scheduler__parsing_processes`
+ The scheduler can run multiple processes in parallel to parse dags. This
defines
+ how many processes will run.
+
+- :ref:`config:scheduler__processor_poll_interval`
+ The number of seconds to wait between consecutive DAG file processing
+
+- :ref:`config:scheduler__schedule_after_task_execution`
+ Should the Task supervisor process perform a “mini scheduler” to attempt to
schedule more tasks of
+ the same DAG. Leaving this on will mean tasks in the same DAG execute
quicker,
+ but might starve out other dags in some circumstances
Review comment:
Don;t know. took it from the reference. I think this might be a bit
different PR to review description of those config variables - as this also
goes into "airflow.cfg" and the .yaml with configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
