Hi Richard,
when using the Kafka journal with default/typical retention times, your
application is responsible for storing snapshots at intervals that are
significantly smaller than the retention time (for example, with a
retention time of 7 days, you may want to take snapshots of your
persistent actors every 3 days or so). Alternatively, configure Kafka to
keep messages "forever" (i.e. set the retention time to the maximum
value) if needed. I don't go into Kafka partitioning details here but it
is possible to implement the journal driver in a way that both a single
persistent actor's data are partitioned *and* kept in order. However,
with the initial implementation, all data for a single persistent actor
must fit on a single Kafka node (different persistent actors are of
course distributed over a Kafka cluster). Hence, deleting old data after
a few weeks and taking snapshots at regular interval is the way to go
(which is good enough for many applications I think).
The real value of the Kafka journal IMO comes with the many external
integrations it supports. For example, you can can use the it as an
input source for Spark streaming
<http://spark.apache.org/docs/latest/streaming-programming-guide.html>
and can do (scalable) stream processing of events generated by
persistent actors i.e. you can easily create Akka -> Kafka -> Spark
Streaming pipelines. This is an alternative to Akka's PersistentView and
even allows processing of events generated by several/all persistent
actors with a single consumer such as a single Spark DStream (which is
currently a limitation <https://github.com/akka/akka/issues/15004> when
using PersistentViews).
I just see this as a starting point for what akka-persistence may
require from all journal implementations in later releases: provide a
persistent event stream generated several persistent actors in a
scalable way. This stream could then be consumed with akka-streams or
Spark Streaming, using a generic connector rather than a
journal-backend-specific, for example.
Initially I just wanted to implement the Kafka integration as
interceptor for journal commands so that events are stored in Kafka in
addition to another journal backend. This may be ok for some projects,
others may think that operational complexity gets too high when you have
to administer a Kafka/Zookeeper cluster in addition to a Cassandra or
MongoDB cluster, for example.
Hope that clarifies things a bit.
Cheers,
Martin
On 12.07.14 15:35, Richard Rodseth wrote:
I saw a tweet from Martin Krasser that he was working on an Akka
Persistence journal plug-in for Kafka. This puzzled me a bit since
Kafka messages are "durable" rather than "persistent" - they are
stored for a configurable time.
Could anyone comment on a typical usage? Assuming that your persistent
actor is going to get recovered before the Kafka topic expires seems odd.
While the Akka/Kafka combination seems great, I always pictured it
would just involve ordinary actors playing the role of Kafka producers
and consumers.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
