> 1 feb 2016 kl. 06:12 skrev 'Martin Krasser' via Akka User List > <[email protected]>: > > > > On 31.01.16 17:48, Roland Kuhn wrote: >> Hi Martin! >> >>> 31 jan 2016 kl. 15:36 skrev 'Martin Krasser' via Akka User List >>> <[email protected] <mailto:[email protected]>>: >>> >>> Hi Roland, >>> >>> a few clarifications/additions inline ... >>> >>> On 31.01.16 10:17, Roland Kuhn wrote: >>>> Hi Paul, >>>> >>>> unfortunately it is impossible to make this “just work”—not least because >>>> you would first have to define what that means. Volker mentioned Eventuate >>>> as a possible solution, but this also is not something that “just works”, >>>> it requires your events to be structured such that your defined state >>>> update functions have all the right properties to make it work. >>> >>> The features and the guarantees provided by Eventuate actually do *not* >>> depend on the structure of events. Eventuate provides means to distinguish >>> causally related from concurrent events and it is just the responsibility >>> of the application to ensure that derived state does not depend on the >>> order of concurrent events. >> >> This is the crux of the matter: ensuring that events are commutative is not >> possible in the general case. I am in complete agreement as to why ACID >> (Associative, Commutative, Idempotent, Distributed) makes sense, but we >> should not misrepresent the fact that these systems are not trivially >> equivalent to strongly consistent systems—it will usually take quite a bit >> of thinking and possibly also adjustments to the business requirements to >> make this work. The result will often be desirable for many reasons, but >> saying that it “just works” is not helpful in my opinion; I’m not saying >> that you implied that, I just want to be very explicit about this part. > > And so are the Eventuate docs :-) Volker either didn't imply that it "just > works", he referred to Eventuate as one option how to deal with issues that > are inevitable when choosing AP over CP. On the other hand, these issues are > often rather easy to manage, provided an API offers the right abstractions > for it. That's at least the experience I made from my projects in 2015.
Yes, good, then we are all in agreement :-) I took the phrase only from Paul’s initial inquiry. > >> >>> Causally related events are always delivered in the same causal order at >>> all locations (datacenters, for example). Relaxing strict order to causal >>> order is what gives you availability and partition-tolerance in >>> multi-datacenter setups. >>> >>>> >>>> Imagine there being two copies of yourself running around and doing >>>> things: it would not be enough for one to tell the other what it has done, >>>> there can be real conflicts that arise from these independent actions >>>> (like one of your selves telling your wife that you love her and shortly >>>> thereafter—without having caught up to that point yet—the other files for >>>> divorce). The key here is coordination, without that certain actions >>>> cannot be taken. And coordination can be impossible, e.g. due to network >>>> partitions, which means that you’ll have to decide whether to be cautious >>>> or reckless. >>> >>> Coordination is just an option. If you want to *prevent* concflicts (you >>> called it being "cautious"), only then you need coordination. Here you give >>> up availability in favor of consistency. The other option is to *allow* >>> conflicts, and resolve them later. This does not require coordination but >>> rather means to track, detect and resolve conflicts. With this option, >>> replicas at different datacenters remain writeable, even if they are >>> partitioned from others (you called that being "reckless"). However, you >>> "apologize" for being "reckless" with conflict resolution :-) A great >>> introductory read on this is Pat Helland's paper Building on Quicksand >>> <http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf>. With that >>> second option, you choose availability over (strong) consistency. >> >> Yes, you’re quoting from my recent presentations on the matter :-) >> >>> The second option is not only relevant for multi-datacenter replication but >>> also more generally for collaboration between (micro)services where you >>> usually don't want to couple the availability of one service to that of >>> others. The price for that is that applications must be able to deal with >>> conflicts, rather than trying to prevent them. A consistent approach for >>> that is often missing in distributed applications and often solved with >>> error-prone ad-hoc solutions. Eventuate tries to change that by not only >>> providing APIs to track, detect, and resolve conflicts in an automated or >>> interactive way but also by providing an infrastructure where distributed >>> services can communicate via events in a causally consistent and reliable >>> way. The underlying event logs give you the following guarantees: >>> >>> - the order of events in a local event log is consistent with causal order >>> i.e. consumers will never see an effect before its cause. >>> - event replication across different locations is reliable and idempotent >>> i.e. consumers will never see duplicates when reading from a local log. >> >> These guarantees are certainly valuable, but I am hesitant to declare >> victory just yet: >> >> Tracking causality in general requires O(n^2) data size where n is the total >> number of participants in a causality chain. This means that it works >> reasonably well for small conversation groups or ephemeral interactions, but >> fails in practice for large networks of interacting agents. > > That's definitely not the case. For tracking potential causality O(n) data > size is required where each participant has an entry in a vector clock > <http://rbmhtechnology.github.io/eventuate/architecture.html#vector-clocks>. > This can be further reduced by using plausible clocks > <https://github.com/RBMHTechnology/eventuate/issues/68> where participants > that share an event log at a given location may also share a clock entry. See > #68 <https://github.com/RBMHTechnology/eventuate/issues/68> and #103 > <https://github.com/RBMHTechnology/eventuate/issues/103> for details. Hmm, I’ll need to swap all those details back in, but I’m pretty sure that when discussing the difference between full causality and E-order Mark Miller came to the conclusion that there are cases that require a vector matrix to be tracked correctly (reading #68 I think we discussed that approach and concluded that it collapses too much of the history, i.e. it leads to pessimistically assuming a lot more causality than there is; plus having a central counter at each location limits scalability of what a location is). Intuitively that also makes sense: the common meaning of the word implies that we have to keep track of who knows what when a message is sent, and “what” is a version vector while “who” takes that one dimension up. And that information needs to be sent with every message. There may be ways to restrict the notion of causality itself or the kind of actions that can be performed in order to reduce this overhead to a single version vector, but even that would need to be sent with every message and therefore over the course of the conversation it would consume quadratic network bandwidth. E-order on the other hand has a constant overhead per message send but restricts the notion of causality to only apply to what is effected by a single source. > >> Having causality alone does not allow your agents to be programmed such that >> they can keep even simple invariants like “do not allow the creation of more >> than 500 blog entries”—if two concurrent histories both create the 500th >> post then conflict resolution becomes a hassle. > > Right, global invariants require coordination which limits availability. > However, this is not in conflict with an AP-by-default approach as > coordination can be added on top where needed. This is the question that needs answering—or that will spur further development: are consistency and availability really that opposed in an absolute sense? And if yes, does it make sense to pick a default, or are these two completely segregated problem spaces? > >> >> The latter point is what users of this abstraction need to consider when >> coming from a strongly consistent RDBMS background, it fundamentally changes >> the world view. The apologies cannot in all cases be contained within the >> system, business processes involving human personnel may need to be created >> to deal with the fallout—this is of course very desirable because otherwise >> these processes would need to be improvised when the hair is on fire (a.k.a. >> the network is split) but it should be mentioned both as a cost and a >> benefit when introducing such a system. >> >> When it comes to interactions spanning multiple microservices I tend to >> favor the Saga approach: creating an external entity that manages the causal >> relationship between the changes effected at different services and the >> failure handling (i.e. apologies) allows the compression of the required >> data for causality tracking—because it is no longer generic but tailored to >> the use-case—as well as the colocation of action and compensating action in >> a single unit. As a bonus these units are highly approachable for >> non-programmers as well because they directly correspond to business >> processes. > > I don't want to enumerate over all the advantages and disadvantages of > central coordination vs. federated collaboration here but central > coordination may be in conflict with availability requirements. Also, central > coordination is often the entry point to monolithic solutions :-) I don’t consider the use of sagas as being related to central coordination, a saga is nothing but an agent just like everything else in the system, it does not possess special powers or represent a central authority—all data are owned by some component and that ownership must always be local where consistency is demanded. This is the point of «Life Beyond Distributed Transactions», and it is the same point Garcia-Molina & Salem make: split up the responsibility so that you can distribute. Foregoing local consistency may be necessary in specific cases, but it makes for a so much harder programming model that I remain unconvinced that it should be recommended often, let alone be the default. One thing that strikes me just now is that the user of Eventuate currently has the responsibility of constructing a correct semi-lattice for their data type—why not use CRDTs to make this safer? > >> >>> Compare this to plain at-least-once based messaging between persistent >>> actors or services (in different datacenters, for example) where you cannot >>> make assumptions on message ordering and duplicates. It usually makes >>> writing correct business logic much harder. Furthermore, if you want to >>> decouple a service from the availability of others you are again faced with >>> the problem of detecting and resolving conflicts. I should mention here >>> that rejecting commands != being available :-). As soon as distributed >>> services/applications shall become more resilient to network partitions, >>> you'll anyway have to deal with many of the issues that Eventuate is >>> already adressing. >>> >>> Eventuate meanwhile emerged from a proof-of-concept in early 2015 to a >>> production ready toolkit for building (globally) distributed, >>> service-oriented CQRS/ES applications. As Eventuate is also almost a >>> functional superset of akka-persistence, I wonder if combining efforts >>> would make sense. Please let me know if this sounds interesting to you. >> >> I am very much interested in attacking this problem, we need to give users >> the tools that reduce their responsibility to the essence of their business >> problem. I must confess that I have not yet studied the Eventuate source >> code, all statements above are based on my incomplete understanding of the >> problem domain and some very interesting conversations at ECOOP last year. >> On this basis my current impression is that while we hold some promising >> pieces in our hands we have not yet found the golden hammer—if it exists at >> all. Perhaps it is time to plan a get-together to assemble the pieces we >> know now and explore the landscape that emerges? > > Good idea. Let's start that initiative. Okay, so how do we go about this? I’m in central Europe (as I believe you are) and would propose a small meeting of people who have an interest in implementing such a system. Who else should participate? Who on this list would want to come? Regards, Roland > >> >> Regards, >> >> Roland >> >>> Regards, >>> Martin >>> >>>> >>>> So, you can use Akka Persistence with the same store in different >>>> locations, but you’ll have to make sure that you don’t emit events to the >>>> same log from different places—there can only be one running source of >>>> truth for each persistenceId at any given time. >>>> >>>> Regards, >>>> >>>> Roland >>>> >>>>> 22 jan 2016 kl. 14:52 skrev Paul Cleary <[email protected] >>>>> <mailto:[email protected]>>: >>>>> >>>>> Will Akka Persistence work if you have two different clusters pointing to >>>>> the same data store? >>>>> >>>>> Imagine I have 2 data centers that point at the same database. >>>>> >>>>> If I have updates happening in both data centers at the same time, will >>>>> akka persistence stomp all over the journal / snapshots? >>>>> >>>>> I know that akka persistence has sequence numbers, and I am not sure how >>>>> those are managed. >>>>> >>>>> It would be great if this just worked, but I am thinking I need to >>>>> implement my own persistence plugin and / or persistence layer in my app >>>>> to make sure that there are no collisions. >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: <http://akka.io/docs/>http://akka.io/docs/ >>>>> >>>>>>>>>> <http://akka.io/docs/> >>>>> >>>>>>>>>> Check the FAQ: >>>>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >>>>> >>>>>>>>>> Search the archives: >>>>> >>>>>>>>>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >>>>> >>>>>>>>>> <https://groups.google.com/group/akka-user> >>>>> --- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to >>>>> <mailto:[email protected]>[email protected] >>>>> <mailto:[email protected]>. >>>>> To post to this group, send email to >>>>> <mailto:[email protected]>[email protected] >>>>> <mailto:[email protected]>. >>>>> Visit this group at >>>>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >>>>> <https://groups.google.com/group/akka-user>. >>>>> For more options, visit >>>>> <https://groups.google.com/d/optout>https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> >>>> Dr. Roland Kuhn >>>> Akka Tech Lead >>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >>>> twitter: @rolandkuhn >>>> <http://twitter.com/#%21/rolandkuhn> >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: <http://akka.io/docs/>http://akka.io/docs/ >>>> >>>>>>>>>> <http://akka.io/docs/> >>>> >>>>>>>>>> Check the FAQ: >>>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >>>> >>>>>>>>>> Search the archives: >>>> >>>>>>>>>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >>>> >>>>>>>>>> <https://groups.google.com/group/akka-user> >>>> --- >>>> You received this message because you are subscribed to the Google Groups >>>> "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to >>>> <mailto:[email protected]>[email protected] >>>> <mailto:[email protected]>. >>>> To post to this group, send email to >>>> <mailto:[email protected]>[email protected] >>>> <mailto:[email protected]>. >>>> Visit this group at >>>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >>>> <https://groups.google.com/group/akka-user>. >>>> For more options, visit >>>> <https://groups.google.com/d/optout>https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> -- >>> Martin Krasser >>> >>> blog: http://krasserm.github.io <http://krasserm.github.io/> >>> code: http://github.com/krasserm <http://github.com/krasserm> >>> twitter: http://twitter.com/mrt1nz <http://twitter.com/mrt1nz> >>> >>> -- >>> >>>>>>>>>> Read the docs: <http://akka.io/docs/>http://akka.io/docs/ >>> >>>>>>>>>> <http://akka.io/docs/> >>> >>>>>>>>>> Check the FAQ: >>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >>> >>>>>>>>>> Search the archives: >>> >>>>>>>>>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >>> >>>>>>>>>> <https://groups.google.com/group/akka-user> >>> --- >>> You received this message because you are subscribed to the Google Groups >>> "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to >>> <mailto:[email protected]>[email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to >>> <mailto:[email protected]>[email protected] >>> <mailto:[email protected]>. >>> Visit this group at >>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >>> <https://groups.google.com/group/akka-user>. >>> For more options, visit >>> <https://groups.google.com/d/optout>https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> >> Dr. Roland Kuhn >> Akka Tech Lead >> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >> twitter: @rolandkuhn >> <http://twitter.com/#%21/rolandkuhn> >> >> -- >> >>>>>>>>>> Read the docs: <http://akka.io/docs/>http://akka.io/docs/ >> >>>>>>>>>> <http://akka.io/docs/> >> >>>>>>>>>> Check the FAQ: >> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> >> >>>>>>>>>> Search the archives: >> >>>>>>>>>> <https://groups.google.com/group/akka-user>https://groups.google.com/group/akka-user >> >>>>>>>>>> <https://groups.google.com/group/akka-user> >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/akka-user >> <https://groups.google.com/group/akka-user>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > -- > Martin Krasser > > blog: http://krasserm.github.io <http://krasserm.github.io/> > code: http://github.com/krasserm <http://github.com/krasserm> > twitter: http://twitter.com/mrt1nz <http://twitter.com/mrt1nz> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ <http://akka.io/docs/> > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > >>>>>>>>>> <https://groups.google.com/group/akka-user> > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/akka-user > <https://groups.google.com/group/akka-user>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. Dr. Roland Kuhn Akka Tech Lead Typesafe <http://typesafe.com/> – Reactive apps on the JVM. twitter: @rolandkuhn <http://twitter.com/#!/rolandkuhn> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
