[ 
https://issues.apache.org/jira/browse/AIRFLOW-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369063#comment-16369063
 ] 

Yuliya Volkova commented on AIRFLOW-1945:
-----------------------------------------

[~mosthege], you can do it with Airflow Plugin way, for example: 
[https://github.com/xnuinside/airflow_plugin_custom_cli/blob/master/custom_cli_plugin.py]
 

> Pass --autoscale to celery workers
> ----------------------------------
>
>                 Key: AIRFLOW-1945
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1945
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: celery, cli
>            Reporter: Michael O.
>            Priority: Trivial
>              Labels: easyfix
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Celery supports autoscaling of the worker pool size (number of tasks that can 
> parallelize within one worker node).  I'd like to propose to support passing 
> the --autoscale parameter to {{airflow worker}}.
> Since this is a trivial change, I am not sure if there's any reason for not 
> being supported already.(?)
> For example
> {{airflow worker --concurrency=4}} will set a fixed pool size of 4.
> With minimal changes in 
> [https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855]
>  it could support
> {{airflow worker --autoscale=2,10}} to set an autoscaled pool size of 2 to 10
> Some references:
> * 
> http://docs.celeryproject.org/en/latest/internals/reference/celery.worker.autoscale.html
> * 
> https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to