Hi, In my scenario, the application sending the message is not waiting for a response, just for an ack signalling that the message is persisted and will be processed. However, this message is sent as part of the (usually transactional) processing of some event in the application, so delaying the ack delays the transaction and handling of that event. This means that it's not feasible for my code to just process the messages and propagate the ack from the final destination, because that could take too long.
Tal On Monday, December 28, 2015 at 1:12:08 PM UTC+2, [email protected] wrote: > > Hi, > Using kafka you have a consumer and a producer api. The producer api sends > a message to kafka and receives an acknowledge. The consumer api read a > message from kafka and the app manually sends the acknowledge to kafka > using this consumer. Generally this acknowledge is send after perform some > processing on the message. Kafka wait receives this acknowledge within a > configurable time, otherwise it will resend the message to consumer. > However for you business requirement I don't know if is enough be > acknowledged the message was persisted into kafka. The external app use > your app in a request-response fashion or in a publish jobs fashion? > Em segunda-feira, 28 de dezembro de 2015 06:27:51 UTC-3, Tal Pressman > escreveu: >> >> Hi, >> >> That's an interesting approach. In your architecture, does Kafka handle >> retrying messages when they fail/timeout? >> Something like that could work for some my use cases, but unfortunately >> not all of them. For example, a common scenario is that my code is actually >> embedded as a JAR inside another application that provides the messages for >> processing via an API. In this case, the application can't (or won't) >> handle retries itself, and it expects the messages to be acknowledged >> reasonably fast (see my 4th requirement above). >> Still, it does give me some directions to consider, thanks! >> >> Tal >> >> >> >> On Sunday, December 27, 2015 at 6:28:01 PM UTC+2, [email protected] >> wrote: >>> >>> Currently I'm developing a microservice architecture and I have >>> requirements simillar to yours. I'm using Kafka and Akka Streams. Kafka is >>> a pub sub infra structure with support to at least once delivery semantic. >>> I use the lib reactive Kafka, that allows consume Kafka messages as an akka >>> stream source. Akka streams has a throttler component, so you may use it to >>> limit rath of incoming messages on your destination in a back-pressured >>> way. You source publish messages in Kafka and you akka system process them >>> in a back-pressured way with at-once-leas-devery. I hope this tips help you. >>> >>> On Sunday, December 27, 2015 at 8:52:01 AM UTC-2, Tal Pressman wrote: >>>> >>>> Hi, >>>> >>>> I'm trying to design a system that reads messages from an external >>>> source, does some processing, and sends them to external destinations, >>>> that >>>> has the following requirements (arranged by priority): >>>> >>>> 1. No message loss. All message processing has to be done using >>>> at-least-once semantics. >>>> 2. Throttling capabilities. Should be able to connect a fast >>>> producer to a slow consumer, without running out of memory. >>>> 3. Low latency (when not back-pressured) between receiving the >>>> message from the source and sending it to the destination. >>>> 4. "Reasonable" latency acknowledging the message to the source. >>>> Reasonable here meaning the system can't just pass the message to the >>>> destination and wait for its ack before acknowledging. >>>> >>>> From 1+2, and also from 4, it appears I need to persist incoming >>>> messages, so I was thinking of using a PersistentActor (possibly with >>>> AtLeastOnceDelivery). But here I get to a conflict between 2 and 3. >>>> >>>> In order to achieve low latency I would prefer to persist the messages >>>> asynchronously, and start processing them without waiting for the >>>> persistence to finish. However, if the destination is I'm back-pressured I >>>> would have to buffer the messages and could run out of memory. Using a >>>> persistence query would take care of the throttling and memory concerns, >>>> but would add too much latency to the "sunny day" scenario. >>>> >>>> All of this leads me to think I need some sort of "hybrid" actor - when >>>> there is no back-pressure it would persist asynchronously, when there is >>>> it >>>> would persist synchronously and read (and process) messages from the >>>> persistence layer. >>>> >>>> So now my actual questions: >>>> >>>> 1. Is there anything I'm missing? Any simple solution to this? Am I >>>> making any wrong assumptions? >>>> 2. Is something like this already implemented (in Akka or some >>>> 3rd-party library)? >>>> 3. I think that other than my last requirement, these are fairly >>>> common requirements. If so, maybe a solution for this scenario should >>>> be >>>> added to Akka? (I'd be willing to do my part and contribute a PR...) >>>> >>>> Thanks, >>>> Tal >>>> >>> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
