On Sun, Jul 13, 2014 at 12:51 AM, Martin Krasser
<[email protected]> wrote:
Hi Richard,
when using the Kafka journal with default/typical
retention times, your application is responsible for
storing snapshots at intervals that are significantly
smaller than the retention time (for example, with a
retention time of 7 days, you may want to take snapshots
of your persistent actors every 3 days or so).
Alternatively, configure Kafka to keep messages "forever"
(i.e. set the retention time to the maximum value) if
needed. I don't go into Kafka partitioning details here
but it is possible to implement the journal driver in a
way that both a single persistent actor's data are
partitioned *and* kept in order. However, with the
initial implementation, all data for a single persistent
actor must fit on a single Kafka node (different
persistent actors are of course distributed over a Kafka
cluster). Hence, deleting old data after a few weeks and
taking snapshots at regular interval is the way to go
(which is good enough for many applications I think).
The real value of the Kafka journal IMO comes with the
many external integrations it supports. For example, you
can can use the it as an input source for Spark streaming
<http://spark.apache.org/docs/latest/streaming-programming-guide.html>
and can do (scalable) stream processing of events
generated by persistent actors i.e. you can easily create
Akka -> Kafka -> Spark Streaming pipelines. This is an
alternative to Akka's PersistentView and even allows
processing of events generated by several/all persistent
actors with a single consumer such as a single Spark
DStream (which is currently a limitation
<https://github.com/akka/akka/issues/15004> when using
PersistentViews).
I just see this as a starting point for what
akka-persistence may require from all journal
implementations in later releases: provide a persistent
event stream generated several persistent actors in a
scalable way. This stream could then be consumed with
akka-streams or Spark Streaming, using a generic
connector rather than a journal-backend-specific, for
example.
Initially I just wanted to implement the Kafka
integration as interceptor for journal commands so that
events are stored in Kafka in addition to another journal
backend. This may be ok for some projects, others may
think that operational complexity gets too high when you
have to administer a Kafka/Zookeeper cluster in addition
to a Cassandra or MongoDB cluster, for example.
Hope that clarifies things a bit.
Cheers,
Martin
On 12.07.14 15:35, Richard Rodseth wrote:
I saw a tweet from Martin Krasser that he was working on
an Akka Persistence journal plug-in for Kafka. This
puzzled me a bit since Kafka messages are "durable"
rather than "persistent" - they are stored for a
configurable time.
Could anyone comment on a typical usage? Assuming that
your persistent actor is going to get recovered before
the Kafka topic expires seems odd.
While the Akka/Kafka combination seems great, I always
pictured it would just involve ordinary actors playing
the role of Kafka producers and consumers.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to
the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to [email protected].
To post to this group, send email to
[email protected].
Visit this group at
http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog:http://krasserm.blogspot.com
code:http://github.com/krasserm
twitter:http://twitter.com/mrt1nz
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to
the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to [email protected].
To post to this group, send email to
[email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the
Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
