vlieven commented on issue #18501: URL: https://github.com/apache/airflow/issues/18501#issuecomment-927584865
Suppose you have a DAG with tasks X -> Y -> Z. Setting task_concurrency=2 and max_active_runs=3 allows you to start task X for dagrun n+2, while task Z is still running in dagruns n and n+0. I'm open to suggestions for better ways to optimize DAG throughput while limiting the number of concurrent tasks though? In any case, I don't think the exact parameter values are relevant to the problem described. I mentioned them to indicate that the problem is not with having no room in the cluster to execute tasks, but rather the tasks not getting scheduled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
