To complement the discussion, Jay Kreps (the author of Kafka) recently posted a very nice paper on precisely the subject of logs as a means to integrate data between systems:
http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying It covers the subject of partition, ordering, state recovery of processors, and more. On Tuesday, November 12, 2013 2:19:08 PM UTC-5, Patrik Nordwall wrote: > > > > 12 nov 2013 kl. 18:01 skrev ahjohannessen <[email protected]<javascript:> > >: > > Hi Patrik, > > The order could, in our case, as well be claim-earnings-2 events followed > by claim-earnings-1 events. > > > Ok, then it can be implemented on top of existing processor building block. > /Patrik > > > I am not sure if there is any real advantage other than perhaps less > wiring around the producers of events. > > One can probably accomplish the same thing by using channels + proxies > that point to > the aggregator actor or similar. > > I am just curious about possible approaches for similar scenarios, e.g. > multiple processors of same type > and one global stream for those :) > > > On Tuesday, November 12, 2013 12:08:04 PM UTC, Patrik Nordwall wrote: >> >> >> >> >> On Tue, Nov 12, 2013 at 12:18 PM, ahjohannessen <[email protected]>wrote: >> >>> Would it make sense to have read model functionality that made it >>> possible >>> to aggregate from 1-n processors of *same* type with different processor >>> ids, e.g: >>> >>> pid = claim-earnings-1 >>> pid = claim-earnings-.. >>> pid = claim-earnings-n >>> >>> In this case there would be no causal dependencies between those >>> streams. >>> >> >> In what order do you expect the events to be replayed? Is it all >> claim-earnings-1 events followed by all claim-earnings-2...? >> >> What is the advantage over having separate processors for each, that >> delegates replayed events to an aggregator actor? >> >> /Patrik >> >> >>> It would >>> be useful functionality for a read model that made it possible to get >>> the total event >>> stream of processors with the same type. >>> >>> >>> Note: Reason for mentioning this is because I am in a situation where we >>> build actor >>> subtree hierarchies up that represent unemployment sessions. Each >>> unemployment >>> session hierarchy might stick around for some months and constitutes a >>> model for >>> computation and data that is used for calculating benefits. >>> A read model that could aggregate streams of processors of same type >>> would >>> probably make some things less complicated in this scenario. >>> >>> >>> On Monday, November 11, 2013 5:00:06 PM UTC, Martin Krasser wrote: >>>> >>>> Hi Andrew, >>>> >>>> there's one additional thing to consider when using read models that >>>> receive events from multiple processors: causal dependency of events. >>>> If >>>> events from different streams causally depend on each other, one has to >>>> make sure that this causal dependency is preserved when replaying >>>> messages. With akka-persistence this can only be achieved (at the >>>> moment) when using a (read/write) processor (instead of the read model) >>>> that journals the live streams it receives. Upon replay, the events are >>>> then received in the very same order. Would akka-persistence replay >>>> streams from the logical journals of the source processors, it could >>>> not >>>> detect causal dependencies. >>>> >>>> From this perspective I'm a bit hesitant to allow a read model to >>>> recover from multiple event streams. Of course, some applications may >>>> define read models for which it doesn't matter if the ordering in the >>>> live stream differs from that in the replayed event stream but this is >>>> a >>>> special case. >>>> >>>> In order to support the general case properly, akka-persistence would >>>> to >>>> need replace sequence numbers by vector clocks (where vector size == >>>> number of processors) and I doubt that this should be the scope of >>>> akka-persistence (a framework on top of it could still add that, if >>>> needed). So I recommend using a plain processor in your case. >>>> >>>> Cheers, >>>> Martin >>>> >>>> On 11.11.13 16:59, Andrew Easter wrote: >>>> > Martin, >>>> > >>>> > This discussion has referred to creating a read only processor with >>>> the same processor id as a read/write counterpart. >>>> > >>>> > This is nice but, at least in the use case I'm looking at, my read >>>> views are interested in receiving updates from multiple processors (each >>>> processor representing an aggregate root of a particular type). >>>> > >>>> > I wonder whether there would have to be some flexibility in the way >>>> messages are routed from journal to read only views? For example, rather >>>> than a straight match on processor id, a regular expression could be used. >>>> In that way, one could choose to add a common prefix (i.e. aggregate root >>>> type) to processors of the same type and have a view that defines a >>>> regular >>>> expression that matches on that prefix (ignoring the unique part of the >>>> id). >>>> > >>>> > Andrew >>>> > >>>> >>>> -- >>>> Martin Krasser >>>> >>>> blog: http://krasserm.blogspot.com >>>> code: http://github.com/krasserm >>>> twitter: http://twitter.com/mrt1nz >>>> >>>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> >> Patrik Nordwall >> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >> Twitter: @patriknw >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected]<javascript:> > . > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/groups/opt_out. > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
