Celery Task Startup Overhead

[Aaron Grubb]

Hi everyone,

I've been testing celery workers with both prefork and eventlet pools and I'm 
noticing massive startup overhead for simple BashOperators. For example, 20x 
instances of:

BashOperator(
    task_id='test0',
    bash_command="echo 'test'",
    dag=dag)

executed concurrently spikes my worker machine to from ~150mb to ~3gb 
(eventlet) or ~3.5gb (prefork) memory and takes ~50 seconds. Is this an 
expected artifact of the 20x python executions or is there some way to reduce 
this?

Thanks,
Aaron