On 19.08.14 10:00, Patrik Nordwall wrote:
On Tue, Aug 19, 2014 at 8:51 AM, Martin Krasser
<[email protected] <mailto:[email protected]>> wrote:
Hi Ashley,
thanks for bringing up these questions. Here are some general
comments:
as you already mentioned (in different words) akka-persistence is
currently optimized around write models rather than read models (=
Q in CQRS) i.e it is optimized for fast, scalable persistence and
recovery of stateful actors (= PersistentActor).
For full CQRS support, the discussions so far (in several other
threads) make the assumption that both write and read models are
backed by the same backend store (assuming read models are
maintained by PersistentView actor, receiving a stream of events
from synthetic or physical "topics").
That is not my view of it, at all. PersistentView is a way to
replicate the events to the read side, which typically will store a
denormalized representation optimized for queries. That query store is
typically not the same as the event store, because the requirements
are very different.
I agree, but recent discussions were about how to join events from
several topics/streams that a PersistentView receives (e.g. all events
of an aggregate type or based on a user-defined join/query). Stores
(journal) that are optimized for high write-throughput are not
necessarily the best choice for serving these joins/queries in an
efficient way. Furthermore, why should I maintain a read model datastore
via
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for example?
Some simple read models may keep this representation in-memory but
that is not what I see as the most common case.
/Patrik
This is a severe limitation, IMO. As Greg already mentioned
elsewhere, some read models may be best backed by a graph
database, for example. Although a graph database may be good for
backing certain read models, it may have limitations for fast
logging of events (something where Kafka or Cassandra are very
good at). Consequently, it definitely makes sense to have
different backend stores for write and read models.
If akka-persistence should have support for CQRS in the future,
its design/API should be extended to allow different backend
stores for write and read models (of course, a provider may choose
to use the same backend store to serve both which may be a
reasonable default). This way PersistentActors log events to one
backend store and PersistentViews (or whatever consumer) generate
read models from the other backend store. Data transfer between
these backend stores can be implementation-specific for
optimization purposes. For example
- Cassandra (for logging events) => Spark (to batch-process logged
events) => Graph database XY (to store events processed with
Spark), or
- Kafka (for logging events) => Spark Streaming (to stream-process
logged events) => Database XY (to store events processed with
Spark Streaming)
- ...
These are just two examples how read model backend stores can be
populated in a highly scalable way (both in batch and streaming
mode). Assuming akka-persistence provides an additional plugin API
for storage backends on the read model side (XY in the examples
above) a wide range of CQRS applications could be covered with
whatever scalability and/or ordering requirements needed by the
respective applications. In case you want to read more about it,
take a look at akka-analytics
<https://github.com/krasserm/akka-analytics> (it is very much work
in progress as I'm waiting for Spark to upgrade to Akka 2.3 and
Kafka to Scala 2.11)
WDYT?
Cheers,
Martin
On 19.08.14 04:52, Ashley Aitken wrote:
I'm keen to hear other people's thoughts on the choice of an
event store for Akka Persistence for doing CQRS.
As mentioned in my other post, I believe that Akka Persistence
only provides part of the story for CQRS (but a very important
part) and that other stores will most likely be needed for query
models (both SQL and NOSQL stores).
Since they are project specific I would like to focus here on
what store is "best" for Akka Persistence for CQRS.
Right now my leading contenders are Kafka and Event Store (but I
haven't thought too much about Cassandra or MongoDB etc). My
knowledge of all of these is limited so please excuse and correct
me if any of my statements are wrong.
KAFKA: Apache Kafka is publish-subscribe messaging rethought as a
distributed commit log.
Persistent topics for publishing and subscribing
Highly scalable and distributed
Need to manually create topics for projections
Each topic has own copy of events
Writing to multiple topics is not atomic
Allows logs to be kept for different amounts of time
Battle tested technology from LinkedIn
Not generally used a lifetime store for events
http://kafka.apache.org
https://github.com/krasserm/akka-persistence-kafka/
EVENT STORE: The open-source, functional database with Complex
Event Processing in JavaScript.
Built specifically for storing and projecting events
Store event once and create virtual projection streams
Journal plugin at early stage in development
Projections are still beta but finalising soon
JSON serialisation (which has +ve and -ve points)
Javascript for projection stream specification
Atom interface helps with debugging
Not as distributed or scalable?
Includes temporal criteria for streams
http://geteventstore.com
https://github.com/EventStore/EventStore.Akka.Persistence
Personally, I like the potential Kafka has to be the event store
/ log for CQRS but also the store for logs in big data processing
and analytics. However, the fact that events need to be manually
replicated to different topics and problems that would be caused
if this wasn't consistent is a worry.
On the other hand, Event Store has been specifically designed and
built for event store and projection processing by a leader in
the field of CQRS. However, it uses a unique set of technologies
and I am not sure of it has been battle tested by many or its
long term viability.
What do others think? What are your thoughts of and has your
experience been with other stores?
MONGODB: ?
CASSANDRA: ?
As mentioned, I can definitely see the use of the last two for
query models in addition to one of the event persistence and
projection stream store but have not really considered them for
the latter myself.
Of course, enormous kudos and no disrespect to any of these
fantastic free and open-source projects
Thanks in advance for sharing any thoughts / experiences.
Cheers,
Ashley.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the
Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog:http://krasserm.blogspot.com
code:http://github.com/krasserm
twitter:http://twitter.com/mrt1nz
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Patrik Nordwall
Typesafe <http://typesafe.com/> - Reactive apps on the JVM
Twitter: @patriknw
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
