Hey folks, I would like to implement an at-least-once ingestion pipeline, say from a source like Kafka, to a datastore, using Akka streams. The method of at-least-once is that each incoming message has an increasing numeric offset, and the idea is that successful writes into the datastore would send back the last committed offset, which should increase over time. If a failure occurs, then the ingestion can be restarted from the last committed offset, by replaying messages from the source (or a write-ahead log, for example).
How would I design such a system? Let's start with the naive implementation: kafkaSource.map(someTransformFunc).to(dataStoreSink) This gives us backpressure, but no ack feedbacks. I can think of two ways to add the ack feedback. One is to make the writing to the datastore not a sink but a mapAsync stage that uses futures to write, and returns the latest completed offset. kafkaSource.map(someTransformFunc).mapAsync(writeToDataStoreFuture _) This is incomplete, because there is no flow. Ideally, the stream of completed offsets would be fed back to kafkaSource so it knows the latest committed offset, and if some error occurred (possibly also a result), then the source could replay messages. How to do this? A second approach, which seems hacky and not so idiomatic, is to make kafkaSource and dataStoreSink into ActorPublisher and ActorSubscribers, respectively. Then, the dataStoreSink could send out of band ack messages to the kafkaSource actor directly. It's not clear to me that backpressure is really needed for the acks, I think the most critical part is that writing to the datastore has backpressure. Any thoughts? Many thanks, Evan -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
