HsiuChuanHsu opened a new issue, #56591: URL: https://github.com/apache/airflow/issues/56591
### Description We are using Apache Airflow with the Celery Executor and need to configure the system to allow tasks in different queues to be executed on specific nodes. > The goal is to achieve fine-grained control over task distribution by assigning queues to designated worker nodes. ### Use case/motivation Assign each queue to run on specific worker nodes to optimize resource allocation and task execution. 1. Celery queue groups for different queues and node assignments 2. Each worker group creates a separate Deployment assigned to specific nodes and queues **Use case**: 1. **Node-specific scheduling**: Route certain queues to specialized node pools (GPU, high-memory, etc.) 2. **High-priority vs. low-priority tasks**: Allocate more resources to critical queues 3. **Resource-intensive workloads**: Dedicate workers with higher memory/CPU to specific queues ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
