Currently I'm developing a microservice architecture and I have requirements simillar to yours. I'm using Kafka and Akka Streams. Kafka is a pub sub infra structure with support to at least once delivery semantic. I use the lib reactive Kafka, that allows consume Kafka messages as an akka stream source. Akka streams has a throttler component, so you may use it to limit rath of incoming messages on your destination in a back-pressured way. You source publish messages in Kafka and you akka system process them in a back-pressured way with at-once-leas-devery. I hope this tips help you.
On Sunday, December 27, 2015 at 8:52:01 AM UTC-2, Tal Pressman wrote: > > Hi, > > I'm trying to design a system that reads messages from an external source, > does some processing, and sends them to external destinations, that has the > following requirements (arranged by priority): > > 1. No message loss. All message processing has to be done using > at-least-once semantics. > 2. Throttling capabilities. Should be able to connect a fast producer > to a slow consumer, without running out of memory. > 3. Low latency (when not back-pressured) between receiving the message > from the source and sending it to the destination. > 4. "Reasonable" latency acknowledging the message to the source. > Reasonable here meaning the system can't just pass the message to the > destination and wait for its ack before acknowledging. > > From 1+2, and also from 4, it appears I need to persist incoming messages, > so I was thinking of using a PersistentActor (possibly with > AtLeastOnceDelivery). But here I get to a conflict between 2 and 3. > > In order to achieve low latency I would prefer to persist the messages > asynchronously, and start processing them without waiting for the > persistence to finish. However, if the destination is I'm back-pressured I > would have to buffer the messages and could run out of memory. Using a > persistence query would take care of the throttling and memory concerns, > but would add too much latency to the "sunny day" scenario. > > All of this leads me to think I need some sort of "hybrid" actor - when > there is no back-pressure it would persist asynchronously, when there is it > would persist synchronously and read (and process) messages from the > persistence layer. > > So now my actual questions: > > 1. Is there anything I'm missing? Any simple solution to this? Am I > making any wrong assumptions? > 2. Is something like this already implemented (in Akka or some > 3rd-party library)? > 3. I think that other than my last requirement, these are fairly > common requirements. If so, maybe a solution for this scenario should be > added to Akka? (I'd be willing to do my part and contribute a PR...) > > Thanks, > Tal > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
