Hi Ashley,
thanks for bringing up these questions. Here are some general comments:
as you already mentioned (in different words) akka-persistence is
currently optimized around write models rather than read models (= Q in
CQRS) i.e it is optimized for fast, scalable persistence and recovery of
stateful actors (= PersistentActor).
For full CQRS support, the discussions so far (in several other threads)
make the assumption that both write and read models are backed by the
same backend store (assuming read models are maintained by
PersistentView actor, receiving a stream of events from synthetic or
physical "topics"). This is a severe limitation, IMO. As Greg already
mentioned elsewhere, some read models may be best backed by a graph
database, for example. Although a graph database may be good for backing
certain read models, it may have limitations for fast logging of events
(something where Kafka or Cassandra are very good at). Consequently, it
definitely makes sense to have different backend stores for write and
read models.
If akka-persistence should have support for CQRS in the future, its
design/API should be extended to allow different backend stores for
write and read models (of course, a provider may choose to use the same
backend store to serve both which may be a reasonable default). This way
PersistentActors log events to one backend store and PersistentViews (or
whatever consumer) generate read models from the other backend store.
Data transfer between these backend stores can be
implementation-specific for optimization purposes. For example
- Cassandra (for logging events) => Spark (to batch-process logged
events) => Graph database XY (to store events processed with Spark), or
- Kafka (for logging events) => Spark Streaming (to stream-process
logged events) => Database XY (to store events processed with Spark
Streaming)
- ...
These are just two examples how read model backend stores can be
populated in a highly scalable way (both in batch and streaming mode).
Assuming akka-persistence provides an additional plugin API for storage
backends on the read model side (XY in the examples above) a wide range
of CQRS applications could be covered with whatever scalability and/or
ordering requirements needed by the respective applications. In case you
want to read more about it, take a look at akka-analytics
<https://github.com/krasserm/akka-analytics> (it is very much work in
progress as I'm waiting for Spark to upgrade to Akka 2.3 and Kafka to
Scala 2.11)
WDYT?
Cheers,
Martin
On 19.08.14 04:52, Ashley Aitken wrote:
I'm keen to hear other people's thoughts on the choice of an event
store for Akka Persistence for doing CQRS.
As mentioned in my other post, I believe that Akka Persistence only
provides part of the story for CQRS (but a very important part) and
that other stores will most likely be needed for query models (both
SQL and NOSQL stores).
Since they are project specific I would like to focus here on what
store is "best" for Akka Persistence for CQRS.
Right now my leading contenders are Kafka and Event Store (but I
haven't thought too much about Cassandra or MongoDB etc). My
knowledge of all of these is limited so please excuse and correct me
if any of my statements are wrong.
KAFKA: Apache Kafka is publish-subscribe messaging rethought as a
distributed commit log.
Persistent topics for publishing and subscribing
Highly scalable and distributed
Need to manually create topics for projections
Each topic has own copy of events
Writing to multiple topics is not atomic
Allows logs to be kept for different amounts of time
Battle tested technology from LinkedIn
Not generally used a lifetime store for events
http://kafka.apache.org
https://github.com/krasserm/akka-persistence-kafka/
EVENT STORE: The open-source, functional database with Complex Event
Processing in JavaScript.
Built specifically for storing and projecting events
Store event once and create virtual projection streams
Journal plugin at early stage in development
Projections are still beta but finalising soon
JSON serialisation (which has +ve and -ve points)
Javascript for projection stream specification
Atom interface helps with debugging
Not as distributed or scalable?
Includes temporal criteria for streams
http://geteventstore.com
https://github.com/EventStore/EventStore.Akka.Persistence
Personally, I like the potential Kafka has to be the event store / log
for CQRS but also the store for logs in big data processing and
analytics. However, the fact that events need to be manually
replicated to different topics and problems that would be caused if
this wasn't consistent is a worry.
On the other hand, Event Store has been specifically designed and
built for event store and projection processing by a leader in the
field of CQRS. However, it uses a unique set of technologies and I am
not sure of it has been battle tested by many or its long term viability.
What do others think? What are your thoughts of and has your
experience been with other stores?
MONGODB: ?
CASSANDRA: ?
As mentioned, I can definitely see the use of the last two for query
models in addition to one of the event persistence and projection
stream store but have not really considered them for the latter myself.
Of course, enormous kudos and no disrespect to any of these fantastic
free and open-source projects
Thanks in advance for sharing any thoughts / experiences.
Cheers,
Ashley.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
