Hi Olger,
installing 10k views/producers won't scale, at least not with the
current implementation. Here are some alternatives:
- Maybe a custom journal plugin is what you need: a plugin that
delegates all write/read requests to the actual journal actor and that
additionally updates a database with the events to be written. This
essentially installs a single "listener" per ActorSystem (this is to
some extend comparable to a database trigger that executes additonal
commands. If the backend datastore supports that directly, I recommend
implementing the trigger there, if possible).
- Instead of having thousands of processors, what speaks against
combining them into a single processor (or only a few) per node?
Further comments inline ...
On 18.04.14 16:10, Olger Warnier wrote:
Hi Martin,
I'm currently working on view composition using the brand new
akka-stream module. Basic idea is to make views stream producers
and to use the akka-stream DSL to merge message/event streams from
several producers into whatever you need. See also
https://twitter.com/mrt1nz/status/457120534111981569
<https://twitter.com/mrt1nz/status/457120534111981569> for a first
running example.
WDYT?
First of all Nice stuff !, I think this is useful for the system at my
hands (real-time patient monitoring based on medical data)
I've seen the streams announcements but did not dive into that yet.
Looking at your code StreamExample.scala it more or less 'clicks' in
concept. (and hopefully in the right way)
From a 'View' perspective as currently is available in
akka-persistence, every producing actor needs a view attached to it in
order to push the events to the streams producer, right ? (when I look
at the ViewProducer.scala code, this is what is done.)
PersistentFlow.fromProcessor("p1").toProducer(materializer)
Now, I have a sharding cluster with an EventsourcedProcessor (expect
10.000ths of these EventsourcedProcessor actor instances) , so I'll
need to create a line like this for every EventsourcedProcessor in
order to get the stream of events together. Thereafter, I need to
merge them together to get a single stream of events. (at least that
is one of the features of using the streams)
Every processor instance itself could create such a producer during
start and send it to another actor that merges received producers.
My goal is to have 'Listeners' (that is my interpretation of a 'View'
due to historic reasons...) that will for instance update a data
store, this will probably happen on on just a few nodes (maybe 1 and
some failover stuff). These 'Listeners' need to attach to the sharded
Eventsourced system and ask to get all event sourced events forwarded.
(publish subscribe more or less).
I wonder if the current View (or ViewProducer) fits this situation due
to the fact you need to create as many views as eventsourcedprocessors
are created.
With the merged streams thereafter, it seems a possibility to have
just one thing per node (I assume actor) that will do the writing to a
data store (not being the eventstore).
What would be the way to get these Views 'automagically' attached to
the proper procesors ?
See above.
And, do you have a pointer how this issue is solved with it's own
eventstore ? In a sharding cluster, you more or less have the same
issue. (would streams change your approach there ?)
There's one journal actor per ActorSystem where n journal actors in a
cluster update a replicated journal
<http://akka.io/community/#plugins_to_akka_persistence>.
Hope that helps.
Cheers,
Martin
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
