Per Celery's documentation <http://docs.celeryproject.org/en/latest/getting-started/brokers/>, Redis and RabbitMQ are the best choices. We use Redis at Airbnb and it's been a breeze, I don't think we've ever had any issues coming from that layer.
<http://docs.celeryproject.org/en/2.4-archived/getting-started/brokers/sqlalchemy.html> Using SqlAlchemy <http://docs.celeryproject.org/en/2.4-archived/getting-started/brokers/sqlalchemy.html> (backed by mysql or postgres) as the Celery broker may work pretty well at the low volume that Airflow typically needs. The Celery docs say something about "can lead to messages being executed multiple times", but Airflow has safeguards for that so it should not be a problem. Airflow has also safeguards around lost messages. I'd love to hear reports of people using that broker in production, let us know if you do! Max On Tue, Jan 3, 2017 at 6:51 AM, Conor Nash <[email protected]> wrote: > We are using Airflow with Python 3.5 and RabbitMQ (CeleryExecutor). We're > using Celery version 3.1.24. For the BROKER_URL connection string we use > "pyamqp://<user>:<pass>@host:5671/vhost" and for CELERY_RESULT_BACKEND we > use "rpc://". > > Is Redis the preferred/better supported backend? > > Cheers, > > Conor Nash > Data Science Consulting > www.conornash.com | @conornash <https://twitter.com/conornash> > > On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman < > [email protected]> wrote: > > > We are using Airflow with Python3, PostgreSQL and Redis; it works just > > fine. > > > > On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke < > > [email protected]> wrote: > > > > > Thanks Dennis. We will perhaps try Redis as there has been a ticket to > > > make librabbitmq Python 3 compatible since 2012. > > > > > > > > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > [email protected]<mailto: > > > [email protected]> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > ________________________________ > > > From: Dennis O'Brien <[email protected]> > > > Sent: 01 December 2016 04:45 > > > To: [email protected] > > > Subject: Re: Python 3 > > > > > > Hi Luke, > > > > > > I've been running Airflow on Python 3.5 with Celery. I'm using Redis > as > > > the message broker (and ElastiCache Redis in production). I haven't > > tried > > > RabbitMQ so I can't speak to its compatibility with Python 3.5. > > > > > > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < > > > [email protected]> wrote: > > > > > > > I thought it would be worth following up on this given our experience > > of > > > > testing Python 3 compatibility. > > > > > > > > > > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for > > interaction > > > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > > > > https://github.com/celery/librabbitmq/issues/13). Has anyone else > been > > > > able to get Airflow/Python 3/Celery working? > > > > > > > > > > > > > > > > Cheers, > > > > Luke Maycock > > > > OLIVER WYMAN > > > > [email protected]<mailto: > > > > [email protected]> > > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > > ________________________________ > > > > From: Maycock, Luke <[email protected]> > > > > Sent: 28 November 2016 12:33 > > > > To: [email protected] > > > > Subject: Re: Python 3 > > > > > > > > Thank you Dmitriy and Li. > > > > > > > > Luke Maycock > > > > OLIVER WYMAN > > > > [email protected]<mailto: > > > > [email protected]> > > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > > ________________________________ > > > > From: Li Xuan Ji <[email protected]> > > > > Sent: 28 November 2016 12:20 > > > > To: [email protected] > > > > Subject: Re: Python 3 > > > > > > > > Airflow is supposed to be Python 3 compatible and any > > > > incompatibilities are bugs on our side. > > > > > > > > I haven't heard of anyone running airflow on windows, but I'm not > sure > > > > what the official stance is. > > > > > > > > On 28 November 2016 at 07:01, Dmitriy Krasnikov < > > [email protected]> > > > > wrote: > > > > > I have been using it and developing plugins for it in Python 3.5 > with > > > no > > > > problems. > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Maycock, Luke [mailto:[email protected] > ] > > > > > Sent: Monday, November 28, 2016 6:58 AM > > > > > To: [email protected] > > > > > Subject: Python 3 > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > I have been using Airflow for a while now and have a couple of > > > questions > > > > that I am hoping someone knows the answer to: > > > > > > > > > > 1. Is Airflow now Python 3 compatible? The documentation used to > > state > > > > that Airflow was only compatible with Python 2.7 (< > > > > > > > https://web.archive.org/web/20160310111640/http:// > > pythonhosted.org/airflow/start.html > > > > > > > > > > > > https://web.archive.org/web/20160502024539/http:// > > pythonhosted.org/airflow/start.html > > > ) > > > > but that statement has since been removed. > > > > > 2. I know Gunicorn does not run on Windows - is this the only > > > > component of Airflow that does not run on Windows? > > > > > > > > > > > > > > > Cheers, > > > > > Luke Maycock > > > > > OLIVER WYMAN > > > > > [email protected]<mailto: > > > > [email protected]> > > > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > ________________________________ > > > > > This e-mail and any attachments may be confidential or legally > > > > privileged. If you received this message in error or are not the > > intended > > > > recipient, you should destroy the e-mail message and any attachments > or > > > > copies, and you are prohibited from retaining, distributing, > disclosing > > > or > > > > using any information contained herein. Please inform us of the > > erroneous > > > > delivery by return e-mail. Thank you for your cooperation. > > > > > > > > > > > > > > > > -- > > > > Im Xuan Ji! > > > > > > > > ________________________________ > > > > This e-mail and any attachments may be confidential or legally > > > privileged. > > > > If you received this message in error or are not the intended > > recipient, > > > > you should destroy the e-mail message and any attachments or copies, > > and > > > > you are prohibited from retaining, distributing, disclosing or using > > any > > > > information contained herein. Please inform us of the erroneous > > delivery > > > by > > > > return e-mail. Thank you for your cooperation. > > > > > > > > ________________________________ > > > > This e-mail and any attachments may be confidential or legally > > > privileged. > > > > If you received this message in error or are not the intended > > recipient, > > > > you should destroy the e-mail message and any attachments or copies, > > and > > > > you are prohibited from retaining, distributing, disclosing or using > > any > > > > information contained herein. Please inform us of the erroneous > > delivery > > > by > > > > return e-mail. Thank you for your cooperation. > > > > > > > > > > ________________________________ > > > This e-mail and any attachments may be confidential or legally > > privileged. > > > If you received this message in error or are not the intended > recipient, > > > you should destroy the e-mail message and any attachments or copies, > and > > > you are prohibited from retaining, distributing, disclosing or using > any > > > information contained herein. Please inform us of the erroneous > delivery > > by > > > return e-mail. Thank you for your cooperation. > > > > > >
