Hi Martin! > 31 jan 2016 kl. 15:36 skrev 'Martin Krasser' via Akka User List > <[email protected]>: > > Hi Roland, > > a few clarifications/additions inline ... > > On 31.01.16 10:17, Roland Kuhn wrote: >> Hi Paul, >> >> unfortunately it is impossible to make this “just work”—not least because >> you would first have to define what that means. Volker mentioned Eventuate >> as a possible solution, but this also is not something that “just works”, it >> requires your events to be structured such that your defined state update >> functions have all the right properties to make it work. > > The features and the guarantees provided by Eventuate actually do *not* > depend on the structure of events. Eventuate provides means to distinguish > causally related from concurrent events and it is just the responsibility of > the application to ensure that derived state does not depend on the order of > concurrent events.
This is the crux of the matter: ensuring that events are commutative is not possible in the general case. I am in complete agreement as to why ACID (Associative, Commutative, Idempotent, Distributed) makes sense, but we should not misrepresent the fact that these systems are not trivially equivalent to strongly consistent systems—it will usually take quite a bit of thinking and possibly also adjustments to the business requirements to make this work. The result will often be desirable for many reasons, but saying that it “just works” is not helpful in my opinion; I’m not saying that you implied that, I just want to be very explicit about this part. > Causally related events are always delivered in the same causal order at all > locations (datacenters, for example). Relaxing strict order to causal order > is what gives you availability and partition-tolerance in multi-datacenter > setups. > >> >> Imagine there being two copies of yourself running around and doing things: >> it would not be enough for one to tell the other what it has done, there can >> be real conflicts that arise from these independent actions (like one of >> your selves telling your wife that you love her and shortly >> thereafter—without having caught up to that point yet—the other files for >> divorce). The key here is coordination, without that certain actions cannot >> be taken. And coordination can be impossible, e.g. due to network >> partitions, which means that you’ll have to decide whether to be cautious or >> reckless. > > Coordination is just an option. If you want to *prevent* concflicts (you > called it being "cautious"), only then you need coordination. Here you give > up availability in favor of consistency. The other option is to *allow* > conflicts, and resolve them later. This does not require coordination but > rather means to track, detect and resolve conflicts. With this option, > replicas at different datacenters remain writeable, even if they are > partitioned from others (you called that being "reckless"). However, you > "apologize" for being "reckless" with conflict resolution :-) A great > introductory read on this is Pat Helland's paper Building on Quicksand > <http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf>. With that > second option, you choose availability over (strong) consistency. Yes, you’re quoting from my recent presentations on the matter :-) > The second option is not only relevant for multi-datacenter replication but > also more generally for collaboration between (micro)services where you > usually don't want to couple the availability of one service to that of > others. The price for that is that applications must be able to deal with > conflicts, rather than trying to prevent them. A consistent approach for that > is often missing in distributed applications and often solved with > error-prone ad-hoc solutions. Eventuate tries to change that by not only > providing APIs to track, detect, and resolve conflicts in an automated or > interactive way but also by providing an infrastructure where distributed > services can communicate via events in a causally consistent and reliable > way. The underlying event logs give you the following guarantees: > > - the order of events in a local event log is consistent with causal order > i.e. consumers will never see an effect before its cause. > - event replication across different locations is reliable and idempotent > i.e. consumers will never see duplicates when reading from a local log. These guarantees are certainly valuable, but I am hesitant to declare victory just yet: Tracking causality in general requires O(n^2) data size where n is the total number of participants in a causality chain. This means that it works reasonably well for small conversation groups or ephemeral interactions, but fails in practice for large networks of interacting agents. Having causality alone does not allow your agents to be programmed such that they can keep even simple invariants like “do not allow the creation of more than 500 blog entries”—if two concurrent histories both create the 500th post then conflict resolution becomes a hassle. The latter point is what users of this abstraction need to consider when coming from a strongly consistent RDBMS background, it fundamentally changes the world view. The apologies cannot in all cases be contained within the system, business processes involving human personnel may need to be created to deal with the fallout—this is of course very desirable because otherwise these processes would need to be improvised when the hair is on fire (a.k.a. the network is split) but it should be mentioned both as a cost and a benefit when introducing such a system. When it comes to interactions spanning multiple microservices I tend to favor the Saga approach: creating an external entity that manages the causal relationship between the changes effected at different services and the failure handling (i.e. apologies) allows the compression of the required data for causality tracking—because it is no longer generic but tailored to the use-case—as well as the colocation of action and compensating action in a single unit. As a bonus these units are highly approachable for non-programmers as well because they directly correspond to business processes. > Compare this to plain at-least-once based messaging between persistent actors > or services (in different datacenters, for example) where you cannot make > assumptions on message ordering and duplicates. It usually makes writing > correct business logic much harder. Furthermore, if you want to decouple a > service from the availability of others you are again faced with the problem > of detecting and resolving conflicts. I should mention here that rejecting > commands != being available :-). As soon as distributed services/applications > shall become more resilient to network partitions, you'll anyway have to deal > with many of the issues that Eventuate is already adressing. > > Eventuate meanwhile emerged from a proof-of-concept in early 2015 to a > production ready toolkit for building (globally) distributed, > service-oriented CQRS/ES applications. As Eventuate is also almost a > functional superset of akka-persistence, I wonder if combining efforts would > make sense. Please let me know if this sounds interesting to you. I am very much interested in attacking this problem, we need to give users the tools that reduce their responsibility to the essence of their business problem. I must confess that I have not yet studied the Eventuate source code, all statements above are based on my incomplete understanding of the problem domain and some very interesting conversations at ECOOP last year. On this basis my current impression is that while we hold some promising pieces in our hands we have not yet found the golden hammer—if it exists at all. Perhaps it is time to plan a get-together to assemble the pieces we know now and explore the landscape that emerges? Regards, Roland > Regards, > Martin > >> >> So, you can use Akka Persistence with the same store in different locations, >> but you’ll have to make sure that you don’t emit events to the same log from >> different places—there can only be one running source of truth for each >> persistenceId at any given time. >> >> Regards, >> >> Roland >> >>> 22 jan 2016 kl. 14:52 skrev Paul Cleary < >>> <mailto:[email protected]>[email protected] >>> <mailto:[email protected]>>: >>> >>> Will Akka Persistence work if you have two different clusters pointing to >>> the same data store? >>> >>> Imagine I have 2 data centers that point at the same database. >>> >>> If I have updates happening in both data centers at the same time, will >>> akka persistence stomp all over the journal / snapshots? >>> >>> I know that akka persistence has sequence numbers, and I am not sure how >>> those are managed. >>> >>> It would be great if this just worked, but I am thinking I need to >>> implement my own persistence plugin and / or persistence layer in my app to >>> make sure that there are no collisions. >>> >>> -- >>> >>>>>>>>>> Read the docs: <http://akka.io/docs/>http://akka.io/docs/ >>> >>>>>>>>>> <http://akka.io/docs/> >>> >>>>>>>>>> Check the FAQ: >>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >>> >>>>>>>>>> <https://groups.google.com/group/akka-user> >>> --- >>> You received this message because you are subscribed to the Google Groups >>> "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to >>> <mailto:[email protected]>[email protected] >>> <mailto:[email protected]>. >>> Visit this group at https://groups.google.com/group/akka-user >>> <https://groups.google.com/group/akka-user>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> >> Dr. Roland Kuhn >> Akka Tech Lead >> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >> twitter: @rolandkuhn >> <http://twitter.com/#%21/rolandkuhn> >> >> -- >> >>>>>>>>>> Read the docs: <http://akka.io/docs/>http://akka.io/docs/ >> >>>>>>>>>> <http://akka.io/docs/> >> >>>>>>>>>> Check the FAQ: >> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >> >>>>>>>>>> Search the archives: >> >>>>>>>>>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >> >>>>>>>>>> <https://groups.google.com/group/akka-user> >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/akka-user >> <https://groups.google.com/group/akka-user>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > -- > Martin Krasser > > blog: http://krasserm.github.io <http://krasserm.github.io/> > code: http://github.com/krasserm <http://github.com/krasserm> > twitter: http://twitter.com/mrt1nz <http://twitter.com/mrt1nz> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ <http://akka.io/docs/> > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > >>>>>>>>>> <https://groups.google.com/group/akka-user> > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/akka-user > <https://groups.google.com/group/akka-user>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. Dr. Roland Kuhn Akka Tech Lead Typesafe <http://typesafe.com/> – Reactive apps on the JVM. twitter: @rolandkuhn <http://twitter.com/#!/rolandkuhn> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
