pbotros opened a new issue #12519: URL: https://github.com/apache/airflow/issues/12519
**Description** Provide the ability to limit task concurrency _per worker_. **Use case / motivation** My use case is that I have a particularly heavy task - one that uses lots of RAM & GPU - where if too many instances of that task are running on the same machine at a time, it'll crash. My ideal situation is to have a flag on the operator, something like `task_concurrency_per_worker`, that'll guarantee at most N instances of a particular task is running on that worker at a time. For example, if I trigger 4 instances of DAG A right now, even with 4 workers and `task_concurrency = 4` on the operator, I believe there's no guarantee that each worker will receive at most 1 instance of the task, and hence it could end up with e.g. 2 instances on worker #1 and 2 instances on worker #2. Another heavy-handed solution would be reducing `worker_concurrency`, but that would restrict worker concurrency for all tasks & DAGs, and so isn't ideal as it's overly restrictive. I primarily work with the CeleryExecutor; I'm not famiiiliar with the other non-local executors to know if this is a reasonable request for those executor types. Thanks! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
