This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v2-5-test by this push:
new 192fd4b540 Update scheduler docs about low priority tasks (#28831)
192fd4b540 is described below
commit 192fd4b540c2a8ddfb3ffce66b024eaab356f3fa
Author: eladkal <[email protected]>
AuthorDate: Wed Jan 11 13:25:31 2023 +0200
Update scheduler docs about low priority tasks (#28831)
Gathered insights from discussion in
https://github.com/apache/airflow/issues/26933 into a paragraph in scheduler
docs to clarify why sometimes low priority tasks are scheduled before high
priority tasks
(cherry picked from commit 493b433ad57088a5f5cabc466c949445e500b4c1)
---
docs/apache-airflow/concepts/scheduler.rst | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/docs/apache-airflow/concepts/scheduler.rst
b/docs/apache-airflow/concepts/scheduler.rst
index 10057c386d..b07c5c16c9 100644
--- a/docs/apache-airflow/concepts/scheduler.rst
+++ b/docs/apache-airflow/concepts/scheduler.rst
@@ -59,6 +59,14 @@ In the UI, it appears as if Airflow is running your tasks a
day **late**
You should refer to :doc:`/dag-run` for details on scheduling a DAG.
+.. note::
+ The scheduler is designed for high throughput. This is an informed design
decision to achieve scheduling
+ tasks as soon as possible. The scheduler checks how many free slots
available in a pool and schedule at most that number of tasks instances in one
iteration.
+ This means that task priority will only come in to effect when there are
more scheduled tasks
+ waiting than the queue slots. Thus there can be cases where low priority
tasks will be schedule before high priority tasks if they share the same batch.
+ For more read about that you can reference `this GitHub discussion
<https://github.com/apache/airflow/discussions/28809>`__.
+
+
DAG File Processing
-------------------
