Thanks for your answer. I'm currently in the middle of something and
haven't had the opportunity to check this out. I'll have a look into your
suggestion ASAP, but it seems it could be the way to go. I'll update this
thread as soon as I test the solution. Thanks!.
BR,
Juanjo.
El lunes, 1 de junio de 2015, 13:32:05 (UTC+2), Akka Team escribió:
>
> Hi Huan,
>
> What you need is a back-channel from the Cassandra Sink to the Kafka
> Source. In Akka Streams the way to solve this is to create an explicit
> backward channel with streams.
>
> For example if you have an async Kafka API that gives back the result of
> your read as a Future, together with some metadata usable for acking, you
> can model it like this (pseudocode, I don't know how Kafka API looks like):
>
> val kafkaReader: Flow[AckId, (MyData, AckId), Unit] =
> Flow[AckId].mapAsync(parallelism = 4) { ack =>
> kafka.acknowledge(ack)
> kafka.readAsync() // Returns Future[(MayData, AckId)]
> }
>
>
> val cassandraWriter: Flow[(MyData, AckId), AckId, Unit] = Flow[(MayData,
> AckId)].mapAsync(parallelism = 4) { case (data, ackid) =>
> cassandra.writeAsync(data).map(_ => ackid) // Returns Future[AckId]
> }
>
> kafkaReader.join(cassandraWriter).run() // Close the loop and start
>
> The above code does not handle failed acknowledgments, you need to add
> that handling yourself (for example instead of sending Ackid from cassandra
> Flow, you send a Success(Ackid) or Failure(AckId)). Also, the above cycle
> will not run, because there are no initial acknowledgments in the loop to
> start it. You can do this by adding 4 dummy acknowledgments:
>
> kafkaReader.join(cassandraWriter ++ Source(List(dummyAck1, ...))).run()
>
> (or you can alternatively concat kafkaReader with a Source that injects 4
> reads from Kafka)
>
> -Endre
>
> On Mon, May 25, 2015 at 8:32 PM, Juan José Vázquez Delgado <
> [email protected] <javascript:>> wrote:
>
>> Hello everyone,
>>
>> I'm starting with Akka Streams and so far everything is going well.
>> However, I have met with a use case that I don't know how to approach. The
>> scenario is a stream with an ActorPublisher as a source that is consuming
>> messages from Kafka and a subscriber as a sink that updates a Cassandra
>> table.
>>
>> Kafka ~> some mapping operations ~> Cassandra
>>
>> The point is that I'd like to explicitly confirm to Kafka every time a
>> message has been successfully processed and inserted into Cassandra so that
>> I could re-read the message in case a disaster happens and the service
>> fails, i.e. some kind of at least once delivery behaviour. How could I
>> approach this in terms of Akka Streams?. Is a supported scenario?.
>>
>> It's true that I always can configure the Kafka consumer with auto-commit
>> behaviour but I'd rather take control of how I'm reading the messages.
>>
>> Thanks in advance for your help.
>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]
>> <javascript:>.
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Akka Team
> Typesafe - Reactive apps on the JVM
> Blog: letitcrash.com
> Twitter: @akkateam
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
