Great work Martin.
On Mon, Jul 14, 2014 at 4:36 PM, Martin Krasser <[email protected]> wrote: > There's now a first release of the Kafka journal. Details at > https://github.com/krasserm/akka-persistence-kafka > > Am Sonntag, 13. Juli 2014 16:19:55 UTC+2 schrieb Martin Krasser: >> >> >> On 13.07.14 16:03, Richard Rodseth wrote: >> >> Thanks for the detailed reply. I might have been forgetting that Akka >> persistence can be used for more than persisting DDD aggregates. I had also >> forgotten that the event store and snapshot store can be different. >> >> >> You can even use Kafka to implement a snapshot store. You just need to >> enable log compaction >> <http://kafka.apache.org/documentation.html#compaction> which will >> always keep the last snapshot (entry) for each persistent actor (key). I >> also plan to implement a snapshot store backed by Kafka but I'm not sure at >> the moment how well Kafka supports large log entries. >> >> >> >> On Sun, Jul 13, 2014 at 12:51 AM, Martin Krasser <[email protected]> >> wrote: >> >>> Hi Richard, >>> >>> when using the Kafka journal with default/typical retention times, your >>> application is responsible for storing snapshots at intervals that are >>> significantly smaller than the retention time (for example, with a >>> retention time of 7 days, you may want to take snapshots of your persistent >>> actors every 3 days or so). Alternatively, configure Kafka to keep messages >>> "forever" (i.e. set the retention time to the maximum value) if needed. I >>> don't go into Kafka partitioning details here but it is possible to >>> implement the journal driver in a way that both a single persistent actor's >>> data are partitioned *and* kept in order. However, with the initial >>> implementation, all data for a single persistent actor must fit on a single >>> Kafka node (different persistent actors are of course distributed over a >>> Kafka cluster). Hence, deleting old data after a few weeks and taking >>> snapshots at regular interval is the way to go (which is good enough for >>> many applications I think). >>> >>> The real value of the Kafka journal IMO comes with the many external >>> integrations it supports. For example, you can can use the it as an input >>> source for Spark streaming >>> <http://spark.apache.org/docs/latest/streaming-programming-guide.html> >>> and can do (scalable) stream processing of events generated by persistent >>> actors i.e. you can easily create Akka -> Kafka -> Spark Streaming >>> pipelines. This is an alternative to Akka's PersistentView and even allows >>> processing of events generated by several/all persistent actors with a >>> single consumer such as a single Spark DStream (which is currently a >>> limitation <https://github.com/akka/akka/issues/15004> when using >>> PersistentViews). >>> >>> I just see this as a starting point for what akka-persistence may >>> require from all journal implementations in later releases: provide a >>> persistent event stream generated several persistent actors in a scalable >>> way. This stream could then be consumed with akka-streams or Spark >>> Streaming, using a generic connector rather than a >>> journal-backend-specific, for example. >>> >>> Initially I just wanted to implement the Kafka integration as >>> interceptor for journal commands so that events are stored in Kafka in >>> addition to another journal backend. This may be ok for some projects, >>> others may think that operational complexity gets too high when you have to >>> administer a Kafka/Zookeeper cluster in addition to a Cassandra or MongoDB >>> cluster, for example. >>> >>> Hope that clarifies things a bit. >>> >>> Cheers, >>> Martin >>> >>> >>> On 12.07.14 15:35, Richard Rodseth wrote: >>> >>> I saw a tweet from Martin Krasser that he was working on an Akka >>> Persistence journal plug-in for Kafka. This puzzled me a bit since Kafka >>> messages are "durable" rather than "persistent" - they are stored for a >>> configurable time. >>> >>> Could anyone comment on a typical usage? Assuming that your persistent >>> actor is going to get recovered before the Kafka topic expires seems odd. >>> >>> While the Akka/Kafka combination seems great, I always pictured it >>> would just involve ordinary actors playing the role of Kafka producers and >>> consumers. >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> Martin Krasser >>> >>> blog: http://krasserm.blogspot.com >>> code: http://github.com/krasserm >>> twitter: http://twitter.com/mrt1nz >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >> current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> Martin Krasser >> >> blog: http://krasserm.blogspot.com >> code: http://github.com/krasserm >> twitter: http://twitter.com/mrt1nz >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- *Jonas Bonér*Phone: +46 733 777 123 Home: jonasboner.com Twitter: @jboner <https://twitter.com/jboner> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
