Hello Jean!
First, the decision is made at Linagora. I hope that we can come up with
enough feedback to convince people in the Apache community.
Why Redis?
- RabbitMQ is a bad match for pub sub, I won't repeat the litany of
issues encountered with it...
- We are using Redis in our tech mix, at least at linagora:
- Backend for our OIDC backchannel logout
- Distributed rate limiter
- Backend for RSpamd rules
- Cost of operation matters to us. We are on a mostly on-premise model
and need to be profitable with ~5-10.000 users which means the money to
pay for the infra bill is limited. We see Redis+Rabbit mix as cheaper to
operate than Pulsar, at least for medium sized deployments.
- We do not master pulsar, so there would be a learning curve.
- Having only the "notification" part of the eventbus on Redis means
we can tolerate unavalability /data loss: it is not that critical.
- Currently the James domain logic would imply one topic per mailbox
(more or less) and we would too quickly IMO end up with an unconfortable
number of topics (million+). A refactoring would be needed.
Not to mention it would ease implementing things like IMAP NOTIFY.
- Timing: I need to provide a HA solution "ASAP" for a production
platform. The "Redis" plan can be implemented in a shorter time frame
than the "pulsar"one.
Note that "queuing" topics are not to be handled with Redis.
Note that all of that makes it a Linagora topic, at least now. If
relevant, we would happily contribute it back ;-)
Is this more clear?
Best regards,
Benoit TELLIER
On 01/03/2024 13:17, Jean Helou wrote:
Hello 👋
I'm not (yet) familiar with the event bus :)
Out of curiosity could you describe what made you chose redis and what
other systems you considered ?
What characteristics you thought would match well with the problem.
I'm interested in part because there is a Jira to implement the event bus
over pulsar so I'd like to take the opportunity to better understand the
design ideas behind the event bus as that would help with the pulsar
implementation.
Jean
Le mer. 28 févr. 2024 à 10:09, Quan tran hong <quan.tranhong1...@gmail.com>
a écrit :
Hi folks,
Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
Event bus user notifications to Redis
<
https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
(TLDR:
we observed some annoying issues with RabbitMQ in a deployment and we think
it could be better to move at least the user notifications part of Event
Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
Redis event bus POC <https://github.com/apache/james-project/pull/2028>.
The POC is considered done to me, and I want to share the POC result to
James devs:
- It is *possible *(the POC worked!) to replace the Event bus user
notifications (key registration) using Redis Pub/Sub. And likely the whole
EventBus (the group registration part as well) can rely on Redis (I did not
do that part).
- I did several performance tests for the POC, and the results were *good*.
Regarding the metrics of Event Bus user notifications, Redis seems to
outperform and show more stability in response time than RabbitMQ. (For
more details, I shared on the PR)
Despite the POC has been worked and shown some prospects, I think we should
monitor it more carefully before applying it to James. Therefore, we
(Linagora) would adopt the Redis Event bus user notifications first in our
TMail project (based on James). We would keep an eye on its stability
before contributing the fine-tuned work toward James.
By the way, it seems someone from the community is interested in building
the whole EventBus using Redis cf
https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
start
for that Redis EventBus.
What do you think about using Redis for EventBus? Would that sound
interesting to you?
Regards,
Quan
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
