GitHub user trikker added a comment to the discussion: Is there a function that allows the celery worker provided by airflow to take DAG from the Rabbitmq Quorum queue?
> > Welcome. I am new to RabbitMQ, but I find that RabbitMQ 3.13 supports > > Quorum Queues, link: https://www.rabbitmq.com/docs/3.13/quorum-queues. I > > can create quorum queues in the web console of RabbitMQ 3.13, so I think > > you don't have to upgrade to RabbitMQ 4.xx to use the Quorum Queues. But we > > need to wait for Airflow to upgrade its Celery version. > >  > > Ah, thanks. You're right—I had forgotten since I've been thinking about this > issue for a long time. As you mentioned, it's true that I don’t necessarily > need to upgrade RabbitMQ to use quorum queues, but there are several reasons > why I want to move to version 4.xx. Currently, I’m using version 3.12.xx, > which is nearing its EOL. In addition, with version 3.xx, there are too many > considerations when setting up clustering, mirroring, and handling network > partitions. I believe quorum queues offer a good solution to these issues, > which is why I’m eager to upgrade to 4.xx. However, starting from 4.xx, > classic queues are no longer supported, so I have to keep 3.xx for > compatibility with Airflow. ok, I know your concern of the problem of classic queues. I care about the HA and correctness of MQ too so I tried to use quorum queues too as soon as I deployed RabbitMQ 3.13. Looking forward to the upgrade of Celery in airflow. GitHub link: https://github.com/apache/airflow/discussions/35649#discussioncomment-11299826 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
