RNHTTR commented on code in PR #25102:
URL: https://github.com/apache/airflow/pull/25102#discussion_r922482060
##########
airflow/config_templates/config.yml:
##########
@@ -62,10 +62,9 @@
default: "SequentialExecutor"
- name: parallelism
description: |
- This defines the maximum number of task instances that can run
concurrently per scheduler in
- Airflow, regardless of the worker count. Generally this value,
multiplied by the number of
- schedulers in your cluster, is the maximum number of task instances
with the running
- state in the metadata database.
+ This defines the maximum number of task instances that can run
concurrently in
+ Airflow, regardless of the worker or scheduler count. This value
represents the maximum number
+ of tasks that can be in the running or queued states in the metadata
database.
Review Comment:
Apologies -- the Airflow deployment I was testing on only had one scheduler,
but I thought it had two. When I updated the scheduler count to 2, double the
amount of tasks could be run in parallel.
However, this doesn't make a lot of sense to me. Why would the number of
schedulers influence a parameter that's used to limit the amount of work that
can be done in parallel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
