Hi, I'm trying to design a system that reads messages from an external source, does some processing, and sends them to external destinations, that has the following requirements (arranged by priority):
1. No message loss. All message processing has to be done using at-least-once semantics. 2. Throttling capabilities. Should be able to connect a fast producer to a slow consumer, without running out of memory. 3. Low latency (when not back-pressured) between receiving the message from the source and sending it to the destination. 4. "Reasonable" latency acknowledging the message to the source. Reasonable here meaning the system can't just pass the message to the destination and wait for its ack before acknowledging. >From 1+2, and also from 4, it appears I need to persist incoming messages, so I was thinking of using a PersistentActor (possibly with AtLeastOnceDelivery). But here I get to a conflict between 2 and 3. In order to achieve low latency I would prefer to persist the messages asynchronously, and start processing them without waiting for the persistence to finish. However, if the destination is I'm back-pressured I would have to buffer the messages and could run out of memory. Using a persistence query would take care of the throttling and memory concerns, but would add too much latency to the "sunny day" scenario. All of this leads me to think I need some sort of "hybrid" actor - when there is no back-pressure it would persist asynchronously, when there is it would persist synchronously and read (and process) messages from the persistence layer. So now my actual questions: 1. Is there anything I'm missing? Any simple solution to this? Am I making any wrong assumptions? 2. Is something like this already implemented (in Akka or some 3rd-party library)? 3. I think that other than my last requirement, these are fairly common requirements. If so, maybe a solution for this scenario should be added to Akka? (I'd be willing to do my part and contribute a PR...) Thanks, Tal -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
