Re: celery problem, AttributeError: async

[Bolke de Bruin]

We will rebranch 1.10 from master. Sorry, I have been too busy with normal life 
to be able to follow up on the release of 1.10.

B.

> On 21 May 2018, at 19:54, Craig Rodrigues  wrote:
> 
> Kaxil,
> 
> Thanks for merging this into master.
> What is the procedure to get this into v1-10-test branch?
> I am heavily testing that branch right now, and want to deploy that branch to 
> my prod system
> this Friday.
> 
> --
> Craig
> 
> On 2018/05/21 14:52:33, Craig Rodrigues  wrote: 
>> I have submitted:
>> 
>> https://github.com/apache/incubator-airflow/pull/3388
>> 
>> --
>> Craig
> 

Re: celery problem, AttributeError: async

[Craig Rodrigues]

I have submitted:

https://github.com/apache/incubator-airflow/pull/3388

--
Craig

On Mon, May 21, 2018 at 7:00 AM Naik Kaxil  wrote:

> Thanks. Please do that.
>
> On 21/05/2018, 14:59, "Craig Rodrigues"  wrote:
>
> celery 4.1.1 was just released last night which has all the async
> problems fixed:
>
> https://github.com/celery/celery/commits/v4.1.1
>
> I'll test this out, and then submit a PR to bump airflow's celery
> version to 4.1.1
>
> --
> Craig
>
> On 2018/05/21 07:20:50, Craig Rodrigues  wrote:
> > Hi,
> >
> > I used a requiremens.txt file with these three lines:
> >
> > git+
> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> > celery>=4.2.0rc3
> > kombu>=4.2.0
> >
> >
> > I did
> >
> > pip install -r requirements.txt
> >
> > When I started my worker, I got:
> >
> > [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor
> CeleryExecutor
> > Traceback (most recent call last):
> >   File "/bin/airflow", line 32, in 
> > args.func(args)
> >   File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line
> 74, in wrapper
> > return f(*args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line
> 959, in worker
> > worker.run(**options)
> >   File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line
> 257, in run
> > **kwargs)
> >   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 101, in __init__
> > self.setup_instance(**self.prepare_args(**kwargs))
> >   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 124, in setup_instance
> > self.should_use_eventloop() if use_eventloop is None
> >   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 243, in should_use_eve
> > self._conninfo.transport.implements.async and
> >   File "/usr/lib/python2.7/site-packages/kombu/transport/base.py",
> line 125, in __getattr__
> > raise AttributeError(key)
> > AttributeError: async
> >
> >
> >
> > I reported this as a bug in celery here:
> >
> > https://github.com/celery/celery/issues/4747
> >
> > It turns out that this is a known problem.  It looks like in kombu,
> they
> > renamed a bunch of uses of "async" to "asynchronous", to not clash
> with the new
> > 'async' keyword in Python 3.7.
> >
> > In celery, they are catching up and doing the same rename.
> > Almost at the same time that I reported this bug, the issue was
> fixed in the master branch of celery:
> >
> >
> https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
> >
> > Hopefully the versions of kombu and celery will stabilize so that it
> all works properly with airflow.
> >
> > --
> > Craig
> >
> >
> >
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>

Re: celery problem, AttributeError: async

[Naik Kaxil]

Thanks. Please do that.

On 21/05/2018, 14:59, "Craig Rodrigues"  wrote:

celery 4.1.1 was just released last night which has all the async problems 
fixed:

https://github.com/celery/celery/commits/v4.1.1

I'll test this out, and then submit a PR to bump airflow's celery version 
to 4.1.1

--
Craig

On 2018/05/21 07:20:50, Craig Rodrigues  wrote: 
> Hi,
> 
> I used a requiremens.txt file with these three lines:
> 
> 
git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> celery>=4.2.0rc3
> kombu>=4.2.0
> 
> 
> I did
> 
> pip install -r requirements.txt
> 
> When I started my worker, I got:
> 
> [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor 
CeleryExecutor
> Traceback (most recent call last):
>   File "/bin/airflow", line 32, in 
> args.func(args)
>   File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, 
in wrapper
> return f(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, 
in worker
> worker.run(**options)
>   File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, 
in run
> **kwargs)
>   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 
101, in __init__
> self.setup_instance(**self.prepare_args(**kwargs))
>   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 
124, in setup_instance
> self.should_use_eventloop() if use_eventloop is None
>   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 
243, in should_use_eve
> self._conninfo.transport.implements.async and
>   File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 
125, in __getattr__
> raise AttributeError(key)
> AttributeError: async
> 
> 
> 
> I reported this as a bug in celery here:
> 
> https://github.com/celery/celery/issues/4747
> 
> It turns out that this is a known problem.  It looks like in kombu, they
> renamed a bunch of uses of "async" to "asynchronous", to not clash with 
the new
> 'async' keyword in Python 3.7.
> 
> In celery, they are catching up and doing the same rename.  
> Almost at the same time that I reported this bug, the issue was fixed in 
the master branch of celery:
> 
> 
https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
> 
> Hopefully the versions of kombu and celery will stabilize so that it all 
works properly with airflow.
> 
> --
> Craig
> 
> 
> 




Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

Re: celery problem, AttributeError: async

[Craig Rodrigues]

celery 4.1.1 was just released last night which has all the async problems 
fixed:

https://github.com/celery/celery/commits/v4.1.1

I'll test this out, and then submit a PR to bump airflow's celery version to 
4.1.1

--
Craig

On 2018/05/21 07:20:50, Craig Rodrigues  wrote: 
> Hi,
> 
> I used a requiremens.txt file with these three lines:
> 
> git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> celery>=4.2.0rc3
> kombu>=4.2.0
> 
> 
> I did
> 
> pip install -r requirements.txt
> 
> When I started my worker, I got:
> 
> [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor 
> CeleryExecutor
> Traceback (most recent call last):
>   File "/bin/airflow", line 32, in 
> args.func(args)
>   File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in 
> wrapper
> return f(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, in 
> worker
> worker.run(**options)
>   File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, in 
> run
> **kwargs)
>   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 101, 
> in __init__
> self.setup_instance(**self.prepare_args(**kwargs))
>   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 124, 
> in setup_instance
> self.should_use_eventloop() if use_eventloop is None
>   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 243, 
> in should_use_eve
> self._conninfo.transport.implements.async and
>   File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 125, 
> in __getattr__
> raise AttributeError(key)
> AttributeError: async
> 
> 
> 
> I reported this as a bug in celery here:
> 
> https://github.com/celery/celery/issues/4747
> 
> It turns out that this is a known problem.  It looks like in kombu, they
> renamed a bunch of uses of "async" to "asynchronous", to not clash with the 
> new
> 'async' keyword in Python 3.7.
> 
> In celery, they are catching up and doing the same rename.  
> Almost at the same time that I reported this bug, the issue was fixed in the 
> master branch of celery:
> 
> https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
> 
> Hopefully the versions of kombu and celery will stabilize so that it all 
> works properly with airflow.
> 
> --
> Craig
> 
> 
> 

celery problem, AttributeError: async

[Craig Rodrigues]

Hi,

I used a requiremens.txt file with these three lines:

git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
celery>=4.2.0rc3
kombu>=4.2.0


I did

pip install -r requirements.txt

When I started my worker, I got:

[2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor CeleryExecutor
Traceback (most recent call last):
  File "/bin/airflow", line 32, in 
args.func(args)
  File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in 
wrapper
return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, in 
worker
worker.run(**options)
  File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, in run
**kwargs)
  File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 101, in 
__init__
self.setup_instance(**self.prepare_args(**kwargs))
  File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 124, in 
setup_instance
self.should_use_eventloop() if use_eventloop is None
  File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 243, in 
should_use_eve
self._conninfo.transport.implements.async and
  File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 125, in 
__getattr__
raise AttributeError(key)
AttributeError: async



I reported this as a bug in celery here:

https://github.com/celery/celery/issues/4747

It turns out that this is a known problem.  It looks like in kombu, they
renamed a bunch of uses of "async" to "asynchronous", to not clash with the new
'async' keyword in Python 3.7.

In celery, they are catching up and doing the same rename.  
Almost at the same time that I reported this bug, the issue was fixed in the 
master branch of celery:

https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad

Hopefully the versions of kombu and celery will stabilize so that it all works 
properly with airflow.

--
Craig