jack created AIRFLOW-3145:
-----------------------------
Summary: reduce airflow dag scheduling latency in production
Key: AIRFLOW-3145
URL: https://issues.apache.org/jira/browse/AIRFLOW-3145
Project: Apache Airflow
Issue Type: Task
Affects Versions: 1.10.1
Reporter: jack
The docs says:
* {{}}
{code:java}
max_threads: Scheduler will spawn multiple threads in parallel to schedule
dags. This is controlled by max_threads with default value of 2. User should
increase this value to a larger value(e.g numbers of cpus where scheduler runs
- 1) in production.{code}
[https://airflow.apache.org/faq.html#how-can-my-airflow-dag-run-faster]
However in the Aiflow.cfg the comment is
{code:java}
#The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run.
max_threads = 2{code}
May I suggest that the
{code:java}
User should increase this value to a larger value(e.g numbers of cpus where
scheduler runs - 1) in production.{code}
will also be written in the comment at Aiflow.cfg
Not everyone read the docs from top to bottom this could be easily missed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
