Thanks for the quick response. Obviously, if message delivery fails then low latency is out of the question. However, in the vast majority of cases, there will be no errors, no message loss, and no back-pressure. And in such cases, I think it should be possible to achieve the lower latency bounds. It would just require some care switching between the 2 states...
Tal On Sunday, December 27, 2015 at 1:50:48 PM UTC+2, rkuhn wrote: > > Without answering concretely I'd like to point out that at-least-once is > fundamentally incompatible with low latency, there is no way around this > conflict. Any middle ground will be a compromise of the kind "at-least-once > but only for X% of the cases", meaning that resolving this conflict is a > genuine business decision. > > Regards, > > Roland > > Sent from my iPhone > > On 27 Dec 2015, at 11:52, Tal Pressman <[email protected] <javascript:>> > wrote: > > Hi, > > I'm trying to design a system that reads messages from an external source, > does some processing, and sends them to external destinations, that has the > following requirements (arranged by priority): > > 1. No message loss. All message processing has to be done using > at-least-once semantics. > 2. Throttling capabilities. Should be able to connect a fast producer > to a slow consumer, without running out of memory. > 3. Low latency (when not back-pressured) between receiving the message > from the source and sending it to the destination. > 4. "Reasonable" latency acknowledging the message to the source. > Reasonable here meaning the system can't just pass the message to the > destination and wait for its ack before acknowledging. > > From 1+2, and also from 4, it appears I need to persist incoming messages, > so I was thinking of using a PersistentActor (possibly with > AtLeastOnceDelivery). But here I get to a conflict between 2 and 3. > > In order to achieve low latency I would prefer to persist the messages > asynchronously, and start processing them without waiting for the > persistence to finish. However, if the destination is I'm back-pressured I > would have to buffer the messages and could run out of memory. Using a > persistence query would take care of the throttling and memory concerns, > but would add too much latency to the "sunny day" scenario. > > All of this leads me to think I need some sort of "hybrid" actor - when > there is no back-pressure it would persist asynchronously, when there is it > would persist synchronously and read (and process) messages from the > persistence layer. > > So now my actual questions: > > 1. Is there anything I'm missing? Any simple solution to this? Am I > making any wrong assumptions? > 2. Is something like this already implemented (in Akka or some > 3rd-party library)? > 3. I think that other than my last requirement, these are fairly > common requirements. If so, maybe a solution for this scenario should be > added to Akka? (I'd be willing to do my part and contribute a PR...) > > Thanks, > Tal > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
