On Sun, Jul 13, 2014 at 12:51 AM, Martin Krasser
<[email protected] <javascript:>> wrote:
Hi Richard,
when using the Kafka journal with default/typical retention
times, your application is responsible for storing snapshots
at intervals that are significantly smaller than the
retention time (for example, with a retention time of 7
days, you may want to take snapshots of your persistent
actors every 3 days or so). Alternatively, configure Kafka
to keep messages "forever" (i.e. set the retention time to
the maximum value) if needed. I don't go into Kafka
partitioning details here but it is possible to implement
the journal driver in a way that both a single persistent
actor's data are partitioned *and* kept in order. However,
with the initial implementation, all data for a single
persistent actor must fit on a single Kafka node (different
persistent actors are of course distributed over a Kafka
cluster). Hence, deleting old data after a few weeks and
taking snapshots at regular interval is the way to go (which
is good enough for many applications I think).
The real value of the Kafka journal IMO comes with the many
external integrations it supports. For example, you can can
use the it as an input source for Spark streaming
<http://spark.apache.org/docs/latest/streaming-programming-guide.html>
and can do (scalable) stream processing of events generated
by persistent actors i.e. you can easily create Akka ->
Kafka -> Spark Streaming pipelines. This is an alternative
to Akka's PersistentView and even allows processing of
events generated by several/all persistent actors with a
single consumer such as a single Spark DStream (which is
currently a limitation
<https://github.com/akka/akka/issues/15004> when using
PersistentViews).
I just see this as a starting point for what
akka-persistence may require from all journal
implementations in later releases: provide a persistent
event stream generated several persistent actors in a
scalable way. This stream could then be consumed with
akka-streams or Spark Streaming, using a generic connector
rather than a journal-backend-specific, for example.
Initially I just wanted to implement the Kafka integration
as interceptor for journal commands so that events are
stored in Kafka in addition to another journal backend. This
may be ok for some projects, others may think that
operational complexity gets too high when you have to
administer a Kafka/Zookeeper cluster in addition to a
Cassandra or MongoDB cluster, for example.
Hope that clarifies things a bit.
Cheers,
Martin
On 12.07.14 15:35, Richard Rodseth wrote:
I saw a tweet from Martin Krasser that he was working on an
Akka Persistence journal plug-in for Kafka. This puzzled me
a bit since Kafka messages are "durable" rather than
"persistent" - they are stored for a configurable time.
Could anyone comment on a typical usage? Assuming that your
persistent actor is going to get recovered before the Kafka
topic expires seems odd.
While the Akka/Kafka combination seems great, I always
pictured it would just involve ordinary actors playing the
role of Kafka producers and consumers.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
<http://doc.akka.io/docs/akka/current/additional/faq.html>
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
<https://groups.google.com/group/akka-user>
---
You received this message because you are subscribed to the
Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to [email protected]
<javascript:>.
To post to this group, send email to
[email protected] <javascript:>.
Visit this group at
http://groups.google.com/group/akka-user
<http://groups.google.com/group/akka-user>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Martin Krasser
blog:http://krasserm.blogspot.com <http://krasserm.blogspot.com/>
code:http://github.com/krasserm
twitter:http://twitter.com/mrt1nz
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
<http://doc.akka.io/docs/akka/current/additional/faq.html>
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
<https://groups.google.com/group/akka-user>
---
You received this message because you are subscribed to the
Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to [email protected]
<javascript:>.
To post to this group, send email to
[email protected] <javascript:>.
Visit this group at http://groups.google.com/group/akka-user
<http://groups.google.com/group/akka-user>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
<http://doc.akka.io/docs/akka/current/additional/faq.html>
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
<https://groups.google.com/group/akka-user>
---
You received this message because you are subscribed to the
Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected] <javascript:>.
To post to this group, send email to [email protected]
<javascript:>.
Visit this group at http://groups.google.com/group/akka-user
<http://groups.google.com/group/akka-user>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
