alexbegg opened a new issue, #33744: URL: https://github.com/apache/airflow/issues/33744
### Apache Airflow version 2.7.0 ### What happened After upgrading to Airflow 2.7.0 in my local environment my Airflow DAGs won't run even with Celery Executor even after changing `celery_app_name` configuration in `celery` section from `airflow.executors.celery_executor` to `airflow.providers.celery.executors.celery_executor`. I see the error actually is unrelated to the recent Airflow Celery provider changes, but is related to Celery's Redis support. What is happening is Airflow fails to send jobs to the worker as the Kombu module is not compatible with Redis 5.0.0 (released last week). It gives this error (I will update this to the full traceback once I can reproduce this error one more time): ``` AttributeError: module 'redis' has no attribute 'client' ``` Celery actually is limiting redis-py to 4.x in an upcoming version of Celery 5.3.x (it is merged to main on August 17, 2023 but it is not yet released: https://github.com/celery/celery/pull/8442 . The latest version is v5.3.1 released on June 18, 2023). Kombu is also going to match Celery and limit redis-py to 4.x in an upcoming version as well (the PR is draft, I am assuming they are waiting for the Celery change to be released: https://github.com/celery/kombu/pull/1776) For now there is not really a way to fix this unless there is a way we can do a redis constraint to avoid 5.x. Or maybe once the next Celery 5.3.x release includes limiting redis-py to 4.x we can possibly limit Celery provider to that version of Celery? ### What you think should happen instead Airflow should be able to send jobs to workers when using Celery Executor ### How to reproduce 1. Start Airflow 2.7.0 with Celery Executor with Redis 5.0.0 installed by default (at the time of this writing) 2. Run a DAG task 3. The scheduler fails to send the job to the worker Workaround: 1. Limit redis-py to 4.x the same way the upcoming release of Celery 5.3.x does, by using this in requirements.txt: `redis>=4.5.2,<5.0.0,!=4.5.5` 2. Start Airflow 2.7.0 with Celery Executor 3. Run a DAG task 4. The task runs successfully ### Operating System Debian GNU/Linux 11 (bullseye) ### Versions of Apache Airflow Providers apache-airflow-providers-celery==3.3.2 ### Deployment Docker-Compose ### Deployment details I am using `bitnami/airflow:2.7.0` image in Docker Compose when I first encountered this issue, but I will test with Breeze as well shortly and then update this issue. ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
