[
https://issues.apache.org/jira/browse/AIRFLOW-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689475#comment-16689475
]
Iuliia Volkova commented on AIRFLOW-1945:
-----------------------------------------
[~ashb], [~Fokko], please close the task, PR was already merged -
https://github.com/apache/incubator-airflow/pull/3989/files
> Pass --autoscale to celery workers
> ----------------------------------
>
> Key: AIRFLOW-1945
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1945
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery, cli
> Reporter: Michael O.
> Assignee: Sai Phanindhra
> Priority: Trivial
> Labels: easyfix
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Celery supports autoscaling of the worker pool size (number of tasks that can
> parallelize within one worker node). I'd like to propose to support passing
> the --autoscale parameter to {{airflow worker}}.
> Since this is a trivial change, I am not sure if there's any reason for not
> being supported already.(?)
> For example
> {{airflow worker --concurrency=4}} will set a fixed pool size of 4.
> With minimal changes in
> [https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855]
> it could support
> {{airflow worker --autoscale=2,10}} to set an autoscaled pool size of 2 to 10
> Some references:
> *
> http://docs.celeryproject.org/en/latest/internals/reference/celery.worker.autoscale.html
> *
> https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
