Thanks Bjorn. On Fri, Jan 30, 2015 at 2:33 AM, Björn Antonsson < [email protected]> wrote:
> Hi Evan, > > I don't think that it's easily possible to feed back things into a Source > right now. You would need a special kind of Source with an input port as > well to wire it up properly in the Graph. The only solution that I can > think of at the moment is to have a PropsSource and a PropsSink that > materialize into actors, and then you "connect" them on the side by sending > normal actor messages. > > B/ > > On 27 January 2015 at 17:08:59, Evan Chan ([email protected]) wrote: > > I got feedback from @ktosopl that you can send feedback in a graph back to > a Source. Does anyone have an example of this? > > Thanks! > > On Tue, Jan 27, 2015 at 8:07 AM, Evan Chan <[email protected]> wrote: > >> Thanks. I know you can form graphs, but doing a custom merge is not >> quite what I'm looking for. >> >> I need to change the state of a Source, using feedback from a Sink. >> >> Example: Kafka source. Depending on Sink output / acknowledgements, reset >> Kafka offset to an earlier offset. >> >> This can't be done by filtering, delaying, etc. the Kafka input source >> via another processing unit. This requires changing the state of the >> Source to begin with, or sending a message to it (if it were Actor-based). >> >> Another example: let's say you have an HTTP-based source. Depending on >> the acknowledgements, we need to give a proper response code: 200, 500, etc. >> >> On Wed, Jan 21, 2015 at 10:36 AM, Martynas Mickevičius < >> [email protected]> wrote: >> >>> Just an idea: you can actually form cycles >>> <http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-graphs.html#Graph_cycles__liveness_and_deadlocks> >>> in >>> the Flow graph, which you can use to feedback acknowledgments. >>> >>> On Sun, Jan 18, 2015 at 2:18 AM, Evan Chan <[email protected]> wrote: >>> >>>> Ping.... anybody? >>>> >>>> >>>> On Thursday, January 15, 2015 at 4:59:19 PM UTC-8, Evan Chan wrote: >>>>> >>>>> Hey folks, >>>>> >>>>> I would like to implement an at-least-once ingestion pipeline, say >>>>> from a source like Kafka, to a datastore, using Akka streams. >>>>> The method of at-least-once is that each incoming message has an >>>>> increasing numeric offset, and the idea is that successful writes into the >>>>> datastore would send back the last committed offset, which should increase >>>>> over time. If a failure occurs, then the ingestion can be restarted from >>>>> the last committed offset, by replaying messages from the source (or a >>>>> write-ahead log, for example). >>>>> >>>>> How would I design such a system? >>>>> >>>>> Let's start with the naive implementation: >>>>> >>>>> kafkaSource.map(someTransformFunc).to(dataStoreSink) >>>>> >>>>> This gives us backpressure, but no ack feedbacks. I can think of two >>>>> ways to add the ack feedback. >>>>> >>>>> One is to make the writing to the datastore not a sink but a mapAsync >>>>> stage that uses futures to write, and returns the latest completed offset. >>>>> >>>>> kafkaSource.map(someTransformFunc).mapAsync(writeToDataStoreFuture _) >>>>> >>>>> This is incomplete, because there is no flow. Ideally, the stream of >>>>> completed offsets would be fed back to kafkaSource so it knows the latest >>>>> committed offset, and if some error occurred (possibly also a result), >>>>> then >>>>> the source could replay messages. How to do this? >>>>> >>>>> A second approach, which seems hacky and not so idiomatic, is to make >>>>> kafkaSource and dataStoreSink into ActorPublisher and ActorSubscribers, >>>>> respectively. Then, the dataStoreSink could send out of band ack messages >>>>> to the kafkaSource actor directly. >>>>> >>>>> It's not clear to me that backpressure is really needed for the acks, >>>>> I think the most critical part is that writing to the datastore has >>>>> backpressure. >>>>> >>>>> Any thoughts? >>>>> >>>>> Many thanks, >>>>> Evan >>>>> >>>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Martynas Mickevičius >>> Typesafe <http://typesafe.com/> – Reactive >>> <http://www.reactivemanifesto.org/> Apps on the JVM >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Akka User List" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/akka-user/wDrW7fMOJIk/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> The fruit of silence is prayer; >> the fruit of prayer is faith; >> the fruit of faith is love; >> the fruit of love is service; >> the fruit of service is peace. -- Mother Teresa >> > > > > -- > The fruit of silence is prayer; > the fruit of prayer is faith; > the fruit of faith is love; > the fruit of love is service; > the fruit of service is peace. -- Mother Teresa > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > -- > Björn Antonsson > Typesafe <http://typesafe.com/> – Reactive Apps on the JVM > twitter: @bantonsson <http://twitter.com/#!/bantonsson> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to a topic in the > Google Groups "Akka User List" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/akka-user/wDrW7fMOJIk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
