o-nikolas commented on issue #46787: URL: https://github.com/apache/airflow/issues/46787#issuecomment-2666732570
Hey folks! Thanks for the submission @insomnes :) This is a neat idea! Being able to throttle how many tasks each executor gets is something I'd like to see for sure. Though, I don't think we should use parallelism for this. That configuration represents _"the maximum number of task instances that can run concurrently per scheduler in Airflow, regardless of the worker count."_ ([link](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism)). This used to be enforced by the executors, but now that we have many that can run concurrently, the scheduler owns this calculation (it really always should have). We _could_ bodge that behaviour in for each executor, but then what would it mean if those values for each executor were less than the total parallelism? Or more? We'd have to add logic to enforce this and ensure it sums to the right amount. I'd much rather see this solved with pools. Being able to assign one (or more) executors to a pool, something that has been a future goal anyway, perfectly fits this use case. In fact this is exactly how we market the use of pools in our docs _"Airflow pools can be used to limit the execution parallelism on arbitrary sets of tasks."_ We really just need to hook executors into that and we'd have a very nice solution. It also avoids having two ways of doing the same thing, which simplifies Airflow as a product. What do folks think of that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
