Without answering concretely I'd like to point out that at-least-once is fundamentally incompatible with low latency, there is no way around this conflict. Any middle ground will be a compromise of the kind "at-least-once but only for X% of the cases", meaning that resolving this conflict is a genuine business decision.
Regards, Roland Sent from my iPhone > On 27 Dec 2015, at 11:52, Tal Pressman <[email protected]> wrote: > > Hi, > > I'm trying to design a system that reads messages from an external source, > does some processing, and sends them to external destinations, that has the > following requirements (arranged by priority): > No message loss. All message processing has to be done using at-least-once > semantics. > Throttling capabilities. Should be able to connect a fast producer to a slow > consumer, without running out of memory. > Low latency (when not back-pressured) between receiving the message from the > source and sending it to the destination. > "Reasonable" latency acknowledging the message to the source. Reasonable here > meaning the system can't just pass the message to the destination and wait > for its ack before acknowledging. > From 1+2, and also from 4, it appears I need to persist incoming messages, so > I was thinking of using a PersistentActor (possibly with > AtLeastOnceDelivery). But here I get to a conflict between 2 and 3. > > In order to achieve low latency I would prefer to persist the messages > asynchronously, and start processing them without waiting for the persistence > to finish. However, if the destination is I'm back-pressured I would have to > buffer the messages and could run out of memory. Using a persistence query > would take care of the throttling and memory concerns, but would add too much > latency to the "sunny day" scenario. > > All of this leads me to think I need some sort of "hybrid" actor - when there > is no back-pressure it would persist asynchronously, when there is it would > persist synchronously and read (and process) messages from the persistence > layer. > > So now my actual questions: > Is there anything I'm missing? Any simple solution to this? Am I making any > wrong assumptions? > Is something like this already implemented (in Akka or some 3rd-party > library)? > I think that other than my last requirement, these are fairly common > requirements. If so, maybe a solution for this scenario should be added to > Akka? (I'd be willing to do my part and contribute a PR...) > Thanks, > Tal > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
