Hello Vikas, We have set up CeleryExecutor with RabbitMQ as messaging broker. In our setup Rabbitmq, airflow scheduler, celery are running on same machine. So far the setup is working fine. One suggestion is avoid using same rabbitmq as "celery_result_backend" as it can choke rabbitmq with persistent results generated by celery. So better to use the airflow datastore for storing celery results as well.
Thanks. On Sat, Oct 8, 2016 at 3:21 AM, Vikas Malhotra <[email protected]> wrote: > Hello Everyone, > > I was wondering if anyone has done setup of Airflow with Celery Executor > with RabbitMQ as backend across EC2 instances and if they are willing to > share their experiences and any pitfalls that I can avoid? > > If anyone has some basic steps or guidelines, that you would be great as > well :) > > Thanks & Regards, > Vikas >
