Michael O. created AIRFLOW-1945:
-----------------------------------
Summary: Pass --autoscale to celery workers
Key: AIRFLOW-1945
URL: https://issues.apache.org/jira/browse/AIRFLOW-1945
Project: Apache Airflow
Issue Type: Improvement
Components: celery, cli
Reporter: Michael O.
Priority: Trivial
Celery supports autoscaling of the worker pool size (number of tasks that can
parallelize within one worker node). I'd like to propose to support passing
the --autoscale parameter to {{airflow worker}}.
Since this is a trivial change, I am not sure if there's any reason for not
being supported already.(?)
For example
{{airflow worker --concurrency=4}} will set a fixed pool size of 4.
With minimal changes in
[https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855]
it could support
{{airflow worker --autoscale=2,10}} to set an autoscaled pool size of 2 to 10
Some references:
*
http://docs.celeryproject.org/en/latest/internals/reference/celery.worker.autoscale.html
*
https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
