o-nikolas commented on code in PR #41107:
URL: https://github.com/apache/airflow/pull/41107#discussion_r1697334371
##########
airflow/config_templates/config.yml:
##########
@@ -89,7 +89,7 @@ core:
This defines the maximum number of task instances that can run
concurrently per scheduler in
Airflow, regardless of the worker count. Generally this value,
multiplied by the number of
schedulers in your cluster, is the maximum number of task instances
with the running
- state in the metadata database.
+ state in the metadata database. Setting this value to zero allows
unlimited parallelism.
Review Comment:
Yeah, I think we should remove it because:
1) It adds unnecessary complexity to the code
2) This behaviour can be achieved by setting parallelism to a sufficiently
high number
3) (most importantly) I think it's actually important for the user to have
to do 2), it forces them to actually think "hmm how parallel _should_ I
actually run tasks? Is infinity appropriated? Will infinity actually cause
degraded performance?". I think allowing 0 gives an easy way for folks to set
and forget.
But I think I may be in the minority, so I did not add any TODOs to remove
it (although I'd love to). So I suppose I'll wait to hear back from others on
that front.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
