My mind must have filtered out the possibility of making snapshots using Views - thanks.
About partitions: I suspected as much. The only thing that I am wondering now is: if it is possible to dynamically create partitions in Kafka? AFAIK the number of partitions is set during topic creation (be it programmatically using API or CLI tools) and there is CLI tool you can use to modify existing topic: https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.AddPartitionTool. To keep the invariant " PersistentActor is the only writer to a partitioned journal topic" you would have to create those partitions dynamically (usually you don't know up front how many PersistentActors your system will have) on per-PersistentActor basis. On the other hand maybe you are assuming that each actor is writing to different topic - but I think this solution is not viable because information about topics is limited by ZK and other factors: http://grokbase.com/t/kafka/users/133v60ng6v/limit-on-number-of-kafka-topic. W dniu wtorek, 26 sierpnia 2014 15:28:47 UTC+2 użytkownik Martin Krasser napisał: > > Hi Andrzej, > > On 26.08.14 09:15, Andrzej Dębski wrote: > > Hello > > Lately I have been reading about a possibility of using Apache Kafka as > journal/snapshot store for akka-persistence. > > I am aware of the plugin created by Martin Krasser: > https://github.com/krasserm/akka-persistence-kafka/ and also I read other > topic about Kafka as journal > https://groups.google.com/forum/#!searchin/akka-user/kakfka/akka-user/iIHmvC6bVrI/zeZJtW0_6FwJ > . > > In both sources I linked two ideas were presented: > > 1. Set log retention to 7 days, take snapshots every 3 days (example > values) > 2. Set log retention to unlimited. > > Here is the first question: in first case wouldn't it mean that > persistent views would receive skewed view of the PersistentActor state > (only events from 7 days) - is it really viable solution? As far as I know > PersistentView can only receive events - it can't receive snapshots from > corresponding PersistentActor (which is good in general case). > > > PersistentViews can create their own snapshots which are isolated from the > corresponding PersistentActor's snapshots. > > > Second question (more directed to Martin): in the thread I linked you > wrote: > > I don't go into Kafka partitioning details here but it is possible to >> implement the journal driver in a way that both a single persistent actor's >> data are partitioned *and* kept in order >> > > I am very interested in this idea. AFAIK it is not yet implemented in > current plugin but I was wondering if you could share high level idea how > would you achieve that (one persistent actor, multiple partitions, ordering > ensured)? > > > The idea is to > > - first write events 1 to n to partition 1 > - then write events n+1 to 2n to partition 2 > - then write events 2n+1 to 3n to partition 3 > - ... and so on > > This works because a PersistentActor is the only writer to a partitioned > journal topic. During replay, you first replay partition 1, then partition > 2 and so on. This should be rather easy to implement in the Kafka journal, > just didn't have time so far; pull requests are welcome :) Btw, the Cassandra > journal <https://github.com/krasserm/akka-persistence-cassandra> follows > the very same strategy for scaling with data volume (by using different > partition keys). > > Cheers, > Martin > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > -- > Martin Krasser > > blog: http://krasserm.blogspot.com > code: http://github.com/krasserm > twitter: http://twitter.com/mrt1nz > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
