Hi, To be honest I found RQ while searching why Airflow isn't currently supporting ActiveMQ instead of RabbitMQ.
I found that RabbitMQ come from a dependency of Celery on Kombu; and I found an incident in Kombu discussing about ActiveMQ support: https://github.com/celery/kombu/issues/548 I asked if ActiveMQ support was still planned in Kombu, and to my surprise I got the hint that some projects have moved from Celery to RQ. Now I realize that RQ means using Redis as the queuing infrastructure, which IHMO is not a so good idea when one realize that it takes 6 nodes to make Redis HA. To sum up, I'm fine if Airflow sticks to Celery, and then I'll see if there's a way to revive interest in adding ActiveMQ support to Kumbu. After all, Airflow and ActiveMQ are Apache projects, so it seems to be fair to look for ActiveMQ support for Airflow to get an Apache-based stack, isn't it ? Kind regards, Alexandre Le jeu. 26 mars 2020 à 18:29, Daniel Imberman <[email protected]> a écrit : > > I mean… we’re not planning it (kind of an “if it’s not broke don’t fix it” > situation), but I don’t think we’re super set on Celery. Would you be > interested in making an AIP to discuss potential benefits? > On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen > <[email protected]>, wrote: > > Hi there, > > > > Looks like couple of projects have moved from Celery to RQ: > > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning > > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq > > > > or are planning such move: > > * https://github.com/getredash/redash/issues/4092 > > > > Is such move considered for Airflow? > > > > Kind regards, > > Alexandre
