RonaldGalea opened a new issue, #34219: URL: https://github.com/apache/airflow/issues/34219
### Description Currently, the Helm [Chart](https://github.com/apache/airflow/blob/main/chart/values.yaml) supports a single Celery worker deployment. The CeleryExecutor however, supports multiple [queues](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html#queues). Ideally, the Helm Chart should allow deploying multiple Celery workers (different images, different resource allocations) to be able to use queues to send tasks to specialised workers (as already highlighted in the [docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html#queues)) ### Use case/motivation It is often the case that tasks might require different hardware resources/different environments. For instance, if I have task A, which requires 1GB of memory, and task B, which requires 10GB, it's inefficient to run them both on the same Celery worker deployment. Having multiple deployments would allow using queues to distribute tasks with finer granularity. ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
