mbaechler commented on pull request #808: URL: https://github.com/apache/james-project/pull/808#issuecomment-1000721926
> Looks awesome! I am eager to try this ;-) Happy you find it interesting. Thank for the review. > Few remarks (feel free to ignore them) > > * Do we have some statistics about how this mail queue perform? It requires a bunch of hardware to do it properly so no. > * It would be **very** nice to have an ADR summurizing this work: why you provided it, what it brings, its architecture, technical details of how Pulsar features are used to implement a mailqueue and the current scope of use (dedicated product...). Maybe with a schema if relevant. Of course. We wanted to finally release something so we didn't aim for perfection. I'm not sure we'll find the motivation to write this doc properly. However, we are eager to share what we did, so I propose we do it another way: we could set up a community meeting to discuss this design and take note of that conversation. It probably won't produce a well structured document but the knowledge sharing would at least be done. WDYT? > > Some architecture remarks from what I could grasp from the review: > > * The remove / browse is done on James side (using an actor model - Akka that do not seem to be distributed) meaning there is implications on the scalability of the size of the queue. The actor is replicated on every James instance. Each time we dequeue a mail, we ask the local instance if it has been deleted. Actors are kept in sync by distributing deletion commands to them using a Pulsar topic. We don't foresee any scalability issue as the complexity is bound to the number of deletion, which should be a small amount anyway. Does it make sense? > * The blob store is used to store arriving messages. Which is similar to what had been done with Rabbit. In my experience, (OVH S3 apis) the creation of such blobs is a lateny pain that can easily take 100s of ms. I wonder if a blobstore like S3 is the right place to store short lived objects from the mail queue. Worth sharing feedbacks but I understand there is no really easy solutions. I don't have more experience than you regarding object storage. We used the abstraction in place as it's not the problem we tried to solve. It could make sense to test an implementation where we put emails directly into Pulsar or Bookeeper ... in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
