Clear! Thanks for the explanation! On Thu, Mar 26, 2020 at 8:08 PM Alexandre Vermeerbergen < [email protected]> wrote:
> We prefer ActiveMQ over RabbitMQ because we are used it deploy it in > production at scale, in HA way, with the upgrade stuff and so on. > Rebuilding all this knowledge with RabbitMQ is an hassle for us... > > Le jeu. 26 mars 2020 à 19:30, Jarek Potiuk <[email protected]> a > écrit : > > > > Out of curiosity why ActiveMQ rather than RabbitMQ (except that it is an > > Apache project that is)? > > > > J, > > > > On Thu, Mar 26, 2020 at 6:40 PM Alexandre Vermeerbergen < > > [email protected]> wrote: > > > > > Hi, > > > > > > To be honest I found RQ while searching why Airflow isn't currently > > > supporting ActiveMQ instead of RabbitMQ. > > > > > > I found that RabbitMQ come from a dependency of Celery on Kombu; and I > > > found an incident in Kombu discussing about ActiveMQ support: > > > https://github.com/celery/kombu/issues/548 > > > > > > I asked if ActiveMQ support was still planned in Kombu, and to my > > > surprise I got the hint that some projects have moved from Celery to > > > RQ. > > > > > > Now I realize that RQ means using Redis as the queuing infrastructure, > > > which IHMO is not a so good idea when one realize that it takes 6 > > > nodes to make Redis HA. > > > > > > To sum up, I'm fine if Airflow sticks to Celery, and then I'll see if > > > there's a way to revive interest in adding ActiveMQ support to Kumbu. > > > After all, Airflow and ActiveMQ are Apache projects, so it seems to be > > > fair to look for ActiveMQ support for Airflow to get an Apache-based > > > stack, isn't it ? > > > > > > Kind regards, > > > Alexandre > > > > > > Le jeu. 26 mars 2020 à 18:29, Daniel Imberman > > > <[email protected]> a écrit : > > > > > > > > I mean… we’re not planning it (kind of an “if it’s not broke don’t > fix > > > it” situation), but I don’t think we’re super set on Celery. Would you > be > > > interested in making an AIP to discuss potential benefits? > > > > On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen < > > > [email protected]>, wrote: > > > > > Hi there, > > > > > > > > > > Looks like couple of projects have moved from Celery to RQ: > > > > > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning > > > > > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq > > > > > > > > > > or are planning such move: > > > > > * https://github.com/getredash/redash/issues/4092 > > > > > > > > > > Is such move considered for Airflow? > > > > > > > > > > Kind regards, > > > > > Alexandre > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
