Thank you Conor! We have managed to get this working after looking at the configuration you provided. We switched out librabbitmq with pyamqp.
Also, thank you to everyone else who responded to this thread! Cheers, Luke Maycock OLIVER WYMAN [email protected]<mailto:[email protected]> www.oliverwyman.com<http://www.oliverwyman.com/> ________________________________ From: Conor Nash <[email protected]> Sent: 03 January 2017 14:51 To: [email protected] Subject: Re: Python 3 We are using Airflow with Python 3.5 and RabbitMQ (CeleryExecutor). We're using Celery version 3.1.24. For the BROKER_URL connection string we use "pyamqp://<user>:<pass>@host:5671/vhost" and for CELERY_RESULT_BACKEND we use "rpc://". Is Redis the preferred/better supported backend? Cheers, Conor Nash Data Science Consulting www.conornash.com | @conornash <https://twitter.com/conornash> On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman < [email protected]> wrote: > We are using Airflow with Python3, PostgreSQL and Redis; it works just > fine. > > On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke < > [email protected]> wrote: > > > Thanks Dennis. We will perhaps try Redis as there has been a ticket to > > make librabbitmq Python 3 compatible since 2012. > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > [email protected]<mailto: > > [email protected]> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > ________________________________ > > From: Dennis O'Brien <[email protected]> > > Sent: 01 December 2016 04:45 > > To: [email protected] > > Subject: Re: Python 3 > > > > Hi Luke, > > > > I've been running Airflow on Python 3.5 with Celery. I'm using Redis as > > the message broker (and ElastiCache Redis in production). I haven't > tried > > RabbitMQ so I can't speak to its compatibility with Python 3.5. > > > > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < > > [email protected]> wrote: > > > > > I thought it would be worth following up on this given our experience > of > > > testing Python 3 compatibility. > > > > > > > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for > interaction > > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > > > https://github.com/celery/librabbitmq/issues/13). Has anyone else been > > > able to get Airflow/Python 3/Celery working? > > > > > > > > > > > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > [email protected]<mailto: > > > [email protected]> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > ________________________________ > > > From: Maycock, Luke <[email protected]> > > > Sent: 28 November 2016 12:33 > > > To: [email protected] > > > Subject: Re: Python 3 > > > > > > Thank you Dmitriy and Li. > > > > > > Luke Maycock > > > OLIVER WYMAN > > > [email protected]<mailto: > > > [email protected]> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > ________________________________ > > > From: Li Xuan Ji <[email protected]> > > > Sent: 28 November 2016 12:20 > > > To: [email protected] > > > Subject: Re: Python 3 > > > > > > Airflow is supposed to be Python 3 compatible and any > > > incompatibilities are bugs on our side. > > > > > > I haven't heard of anyone running airflow on windows, but I'm not sure > > > what the official stance is. > > > > > > On 28 November 2016 at 07:01, Dmitriy Krasnikov < > [email protected]> > > > wrote: > > > > I have been using it and developing plugins for it in Python 3.5 with > > no > > > problems. > > > > > > > > > > > > -----Original Message----- > > > > From: Maycock, Luke [mailto:[email protected]] > > > > Sent: Monday, November 28, 2016 6:58 AM > > > > To: [email protected] > > > > Subject: Python 3 > > > > > > > > Hi All, > > > > > > > > > > > > I have been using Airflow for a while now and have a couple of > > questions > > > that I am hoping someone knows the answer to: > > > > > > > > 1. Is Airflow now Python 3 compatible? The documentation used to > state > > > that Airflow was only compatible with Python 2.7 (< > > > > > https://web.archive.org/web/20160310111640/http:// > pythonhosted.org/airflow/start.html > > > > > > > > > https://web.archive.org/web/20160502024539/http:// > pythonhosted.org/airflow/start.html > > ) > > > but that statement has since been removed. > > > > 2. I know Gunicorn does not run on Windows - is this the only > > > component of Airflow that does not run on Windows? > > > > > > > > > > > > Cheers, > > > > Luke Maycock > > > > OLIVER WYMAN > > > > [email protected]<mailto: > > > [email protected]> > > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > ________________________________ > > > > This e-mail and any attachments may be confidential or legally > > > privileged. If you received this message in error or are not the > intended > > > recipient, you should destroy the e-mail message and any attachments or > > > copies, and you are prohibited from retaining, distributing, disclosing > > or > > > using any information contained herein. Please inform us of the > erroneous > > > delivery by return e-mail. Thank you for your cooperation. > > > > > > > > > > > > -- > > > Im Xuan Ji! > > > > > > ________________________________ > > > This e-mail and any attachments may be confidential or legally > > privileged. > > > If you received this message in error or are not the intended > recipient, > > > you should destroy the e-mail message and any attachments or copies, > and > > > you are prohibited from retaining, distributing, disclosing or using > any > > > information contained herein. Please inform us of the erroneous > delivery > > by > > > return e-mail. Thank you for your cooperation. > > > > > > ________________________________ > > > This e-mail and any attachments may be confidential or legally > > privileged. > > > If you received this message in error or are not the intended > recipient, > > > you should destroy the e-mail message and any attachments or copies, > and > > > you are prohibited from retaining, distributing, disclosing or using > any > > > information contained herein. Please inform us of the erroneous > delivery > > by > > > return e-mail. Thank you for your cooperation. > > > > > > > ________________________________ > > This e-mail and any attachments may be confidential or legally > privileged. > > If you received this message in error or are not the intended recipient, > > you should destroy the e-mail message and any attachments or copies, and > > you are prohibited from retaining, distributing, disclosing or using any > > information contained herein. Please inform us of the erroneous delivery > by > > return e-mail. Thank you for your cooperation. > > > ________________________________ This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
