Hi Martin, I agree, streams should be a first class citizen in akka persistence. I'm afk right now, could you ensure that this is adequately represented in the issue tracker?
Thanks, Roland On July 16, 2014 8:08:57 AM CEST, Martin Krasser <[email protected]> wrote: > >On 16.07.14 05:25, Ashley Aitken wrote: >> >> I think this is a fantastic development (if I understand it >correctly). >> >> From my reading and basic understanding I had concerns about the need > >> for two event infrastructures to 1) implement more complex view >models >> in CQRS because currently Views can't follow more than one >> PersistentActor, and 2) integration of events with non-Akka based >systems. >> >> Please correct me if I am wrong but having one powerful event >> infrastructure (like Kafka) as the "event store" to use across >> applications will enable (2) and possibly (1) for now > >At the moment, it is not possible that a PersistentView consumes from a > >user-defined topic ><https://github.com/krasserm/akka-persistence-kafka#user-defined-topics> > >(user-defined event stream), this requires an extension to Akka >Persistence (both SPI and API). I think it makes sense to go into that >direction and to require that all journal backend stores support that, >so that user-defined event streams can be consumed both Akka-internally > >and by external consumers (that directly connect to the backend store) >as well. > >> and in the future perhaps with akka-streams. > >There already a PersistentView -> akka-streams integration ><http://doc.akka.io/api/akka-stream-and-http-experimental/0.4/#akka.persistence.stream.PersistentFlow$>. > >Once PersistentViews can consume user-defined event streams, you can >automatically expose them as Akka streams. > >> Particularly as Kafka provides publish-subscribe functionality. >> >> Event stores and streams seem so central to many contemporary >systems. >> >> >> On Monday, 14 July 2014 23:08:08 UTC+8, Martin Krasser wrote: >> >> Thanks Heiko, really hope to get some user feedback ... >> >> On 14.07.14 17:04, Heiko Seeberger wrote: >>> Fantastic! >>> >>> Great work, Martin. Keep it coming! >>> >>> Heiko >>> >>> On 14 Jul 2014, at 16:36, Martin Krasser <[email protected] >>> <javascript:>> wrote: >>> >>>> There's now a first release of the Kafka journal. Details at >>>> https://github.com/krasserm/akka-persistence-kafka >>>> <https://github.com/krasserm/akka-persistence-kafka> >>>> >>>> Am Sonntag, 13. Juli 2014 16:19:55 UTC+2 schrieb Martin >Krasser: >>>> >>>> >>>> On 13.07.14 16:03, Richard Rodseth wrote: >>>>> Thanks for the detailed reply. I might have been >forgetting >>>>> that Akka persistence can be used for more than persisting >>>>> DDD aggregates. I had also forgotten that the event store >>>>> and snapshot store can be different. >>>> >>>> You can even use Kafka to implement a snapshot store. You >>>> just need to enable log compaction >>>> <http://kafka.apache.org/documentation.html#compaction> >>>> which will always keep the last snapshot (entry) for each >>>> persistent actor (key). I also plan to implement a snapshot >>>> store backed by Kafka but I'm not sure at the moment how >>>> well Kafka supports large log entries. >>>> >>>>> >>>>> >>>>> On Sun, Jul 13, 2014 at 12:51 AM, Martin Krasser >>>>> <[email protected]> wrote: >>>>> >>>>> Hi Richard, >>>>> >>>>> when using the Kafka journal with default/typical >>>>> retention times, your application is responsible for >>>>> storing snapshots at intervals that are significantly >>>>> smaller than the retention time (for example, with a >>>>> retention time of 7 days, you may want to take >>>>> snapshots of your persistent actors every 3 days or >>>>> so). Alternatively, configure Kafka to keep messages >>>>> "forever" (i.e. set the retention time to the maximum >>>>> value) if needed. I don't go into Kafka partitioning >>>>> details here but it is possible to implement the >>>>> journal driver in a way that both a single persistent >>>>> actor's data are partitioned *and* kept in order. >>>>> However, with the initial implementation, all data for >>>>> a single persistent actor must fit on a single Kafka >>>>> node (different persistent actors are of course >>>>> distributed over a Kafka cluster). Hence, deleting old >>>>> data after a few weeks and taking snapshots at regular >>>>> interval is the way to go (which is good enough for >>>>> many applications I think). >>>>> >>>>> The real value of the Kafka journal IMO comes with the >>>>> many external integrations it supports. For example, >>>>> you can can use the it as an input source for Spark >>>>> streaming >>>>> ><http://spark.apache.org/docs/latest/streaming-programming-guide.html> >>>>> and can do (scalable) stream processing of events >>>>> generated by persistent actors i.e. you can easily >>>>> create Akka -> Kafka -> Spark Streaming pipelines. >This >>>>> is an alternative to Akka's PersistentView and even >>>>> allows processing of events generated by several/all >>>>> persistent actors with a single consumer such as a >>>>> single Spark DStream (which is currently a limitation >>>>> <https://github.com/akka/akka/issues/15004> when using >>>>> PersistentViews). >>>>> >>>>> I just see this as a starting point for what >>>>> akka-persistence may require from all journal >>>>> implementations in later releases: provide a >persistent >>>>> event stream generated several persistent actors in a >>>>> scalable way. This stream could then be consumed with >>>>> akka-streams or Spark Streaming, using a generic >>>>> connector rather than a journal-backend-specific, for >>>>> example. >>>>> >>>>> Initially I just wanted to implement the Kafka >>>>> integration as interceptor for journal commands so >that >>>>> events are stored in Kafka in addition to another >>>>> journal backend. This may be ok for some projects, >>>>> others may think that operational complexity gets too >>>>> high when you have to administer a Kafka/Zookeeper >>>>> cluster in addition to a Cassandra or MongoDB cluster, >>>>> for example. >>>>> >>>>> Hope that clarifies things a bit. >>>>> >>>>> Cheers, >>>>> Martin >>>>> >>>>> >>>>> On 12.07.14 15:35, Richard Rodseth wrote: >>>>>> I saw a tweet from Martin Krasser that he was working >>>>>> on an Akka Persistence journal plug-in for Kafka. >This >>>>>> puzzled me a bit since Kafka messages are "durable" >>>>>> rather than "persistent" - they are stored for a >>>>>> configurable time. >>>>>> >>>>>> Could anyone comment on a typical usage? Assuming >that >>>>>> your persistent actor is going to get recovered >before >>>>>> the Kafka topic expires seems odd. >>>>>> >>>>>> While the Akka/Kafka combination seems great, I >always >>>>>> pictured it would just involve ordinary actors >playing >>>>>> the role of Kafka producers and consumers. >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: >>>>>> >http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>> ><http://doc.akka.io/docs/akka/current/additional/faq.html> >>>>>> >>>>>>>>>> Search the archives: >>>>>> https://groups.google.com/group/akka-user >>>>>> <https://groups.google.com/group/akka-user> >>>>>> --- >>>>>> You received this message because you are subscribed >>>>>> to the Google Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving >>>>>> emails from it, send an email to >>>>>> [email protected]. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> Visit this group at >>>>>> http://groups.google.com/group/akka-user >>>>>> <http://groups.google.com/group/akka-user>. >>>>>> For more options, visit >>>>>> https://groups.google.com/d/optout >>>>>> <https://groups.google.com/d/optout>. >>>>> >>>>> -- >>>>> Martin Krasser >>>>> >>>>> blog:http://krasserm.blogspot.com ><http://krasserm.blogspot.com/> >>>>> code:http://github.com/krasserm >>>>> twitter:http://twitter.com/mrt1nz >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> >http://doc.akka.io/docs/akka/current/additional/faq.html ><http://doc.akka.io/docs/akka/current/additional/faq.html> >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> <https://groups.google.com/group/akka-user> >>>>> --- >>>>> You received this message because you are subscribed >to >>>>> the Google Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving >>>>> emails from it, send an email to >>>>> [email protected]. >>>>> To post to this group, send email to >>>>> [email protected]. >>>>> Visit this group at >>>>> http://groups.google.com/group/akka-user >>>>> <http://groups.google.com/group/akka-user>. >>>>> For more options, visit >>>>> https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>>> >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> <https://groups.google.com/group/akka-user> >>>>> --- >>>>> You received this message because you are subscribed to >the >>>>> Google Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails >>>>> from it, send an email to [email protected]. >>>>> To post to this group, send email to >[email protected]. >>>>> Visit this group at >>>>> http://groups.google.com/group/akka-user >>>>> <http://groups.google.com/group/akka-user>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> -- >>>> Martin Krasser >>>> >>>> blog:http://krasserm.blogspot.com ><http://krasserm.blogspot.com/> >>>> code:http://github.com/krasserm >>>> twitter:http://twitter.com/mrt1nz >>>> >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> <https://groups.google.com/group/akka-user> >>>> --- >>>> You received this message because you are subscribed to the >>>> Google Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from >>>> it, send an email to [email protected] ><javascript:>. >>>> To post to this group, send email to [email protected] >>>> <javascript:>. >>>> Visit this group at http://groups.google.com/group/akka-user >>>> <http://groups.google.com/group/akka-user>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> >>> Heiko Seeberger >>> Twitter: @hseeberger >>> Web: heikoseeberger.de <http://heikoseeberger.de> >>> >>> >>> >>> >> >> -- >> Martin Krasser >> >> blog:http://krasserm.blogspot.com >> code:http://github.com/krasserm >> twitter:http://twitter.com/mrt1nz >> > >-- >Martin Krasser > >blog: http://krasserm.blogspot.com >code: http://github.com/krasserm >twitter: http://twitter.com/mrt1nz > >-- >>>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>> Check the FAQ: >http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>> Search the archives: >https://groups.google.com/group/akka-user >--- >You received this message because you are subscribed to the Google >Groups "Akka User List" group. >To unsubscribe from this group and stop receiving emails from it, send >an email to [email protected]. >To post to this group, send email to [email protected]. >Visit this group at http://groups.google.com/group/akka-user. >For more options, visit https://groups.google.com/d/optout. Regards, Dr. Roland Kuhn Akka Tech Lead Typesafe – Reactive apps on the JVM twitter: @rolandkuhn -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
