On 31.01.16 17:48, Roland Kuhn wrote:
Hi Martin!
31 jan 2016 kl. 15:36 skrev 'Martin Krasser' via Akka User List
<[email protected] <mailto:[email protected]>>:
Hi Roland,
a few clarifications/additions inline ...
On 31.01.16 10:17, Roland Kuhn wrote:
Hi Paul,
unfortunately it is impossible to make this “just work”—not least
because you would first have to define what that means. Volker
mentioned Eventuate as a possible solution, but this also is not
something that “just works”, it requires your events to be
structured such that your defined state update functions have all
the right properties to make it work.
The features and the guarantees provided by Eventuate actually do
*not* depend on the structure of events. Eventuate provides means to
distinguish causally related from concurrent events and it is just
the responsibility of the application to ensure that derived state
does not depend on the order of concurrent events.
This is the crux of the matter: ensuring that events are commutative
is not possible in the general case. I am in complete agreement as to
why ACID (Associative, Commutative, Idempotent, Distributed) makes
sense, but we should not misrepresent the fact that these systems are
not trivially equivalent to strongly consistent systems—it will
usually take quite a bit of thinking and possibly also adjustments to
the business requirements to make this work. The result will often be
desirable for many reasons, but saying that it “just works” is not
helpful in my opinion; I’m not saying that you implied that, I just
want to be very explicit about this part.
And so are the Eventuate docs :-) Volker either didn't imply that it
"just works", he referred to Eventuate as one option how to deal with
issues that are inevitable when choosing AP over CP. On the other hand,
these issues are often rather easy to manage, provided an API offers the
right abstractions for it. That's at least the experience I made from my
projects in 2015.
Causally related events are always delivered in the same causal order
at all locations (datacenters, for example). Relaxing strict order to
causal order is what gives you availability and partition-tolerance
in multi-datacenter setups.
Imagine there being two copies of yourself running around and doing
things: it would not be enough for one to tell the other what it has
done, there can be real conflicts that arise from these independent
actions (like one of your selves telling your wife that you love her
and shortly thereafter—without having caught up to that point
yet—the other files for divorce). The key here is coordination,
without that certain actions cannot be taken. And coordination can
be impossible, e.g. due to network partitions, which means that
you’ll have to decide whether to be cautious or reckless.
Coordination is just an option. If you want to *prevent* concflicts
(you called it being "cautious"), only then you need coordination.
Here you give up availability in favor of consistency. The other
option is to *allow* conflicts, and resolve them later. This does not
require coordination but rather means to track, detect and resolve
conflicts. With this option, replicas at different datacenters remain
writeable, even if they are partitioned from others (you called that
being "reckless"). However, you "apologize" for being "reckless" with
conflict resolution :-) A great introductory read on this is Pat
Helland's paperBuilding on Quicksand
<http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf>. With
that second option, you choose availability over (strong) consistency.
Yes, you’re quoting from my recent presentations on the matter :-)
The second option is not only relevant for multi-datacenter
replication but also more generally for collaboration between
(micro)services where you usually don't want to couple the
availability of one service to that of others. The price for that is
that applications must be able to deal with conflicts, rather than
trying to prevent them. A consistent approach for that is often
missing in distributed applications and often solved with error-prone
ad-hoc solutions. Eventuate tries to change that by not only
providing APIs to track, detect, and resolve conflicts in an
automated or interactive way but also by providing an infrastructure
where distributed services can communicate via events in a causally
consistent and reliable way. The underlying event logs give you the
following guarantees:
- the order of events in a local event log is consistent with causal
order i.e. consumers will never see an effect before its cause.
- event replication across different locations is reliable and
idempotent i.e. consumers will never see duplicates when reading from
a local log.
These guarantees are certainly valuable, but I am hesitant to declare
victory just yet:
* Tracking causality in general requires O(n^2) data size where n is
the total number of participants in a causality chain. This means
that it works reasonably well for small conversation groups or
ephemeral interactions, but fails in practice for large networks
of interacting agents.
That's definitely not the case. For tracking potential causality O(n)
data size is required where each participant has an entry in a vector
clock
<http://rbmhtechnology.github.io/eventuate/architecture.html#vector-clocks>.
This can be further reduced by using plausible clocks
<https://github.com/RBMHTechnology/eventuate/issues/68> where
participants that share an event log at a given location may also share
a clock entry. See #68
<https://github.com/RBMHTechnology/eventuate/issues/68> and #103
<https://github.com/RBMHTechnology/eventuate/issues/103> for details.
* Having causality alone does not allow your agents to be programmed
such that they can keep even simple invariants like “do not allow
the creation of more than 500 blog entries”—if two concurrent
histories both create the 500th post then conflict resolution
becomes a hassle.
Right, global invariants require coordination which limits availability.
However, this is not in conflict with an AP-by-default approach as
coordination can be added on top where needed.
The latter point is what users of this abstraction need to consider
when coming from a strongly consistent RDBMS background, it
fundamentally changes the world view. The apologies cannot in all
cases be contained within the system, business processes involving
human personnel may need to be created to deal with the fallout—this
is of course very desirable because otherwise these processes would
need to be improvised when the hair is on fire (a.k.a. the network is
split) but it should be mentioned both as a cost and a benefit when
introducing such a system.
When it comes to interactions spanning multiple microservices I tend
to favor the Saga approach: creating an external entity that manages
the causal relationship between the changes effected at different
services and the failure handling (i.e. apologies) allows the
compression of the required data for causality tracking—because it is
no longer generic but tailored to the use-case—as well as the
colocation of action and compensating action in a single unit. As a
bonus these units are highly approachable for non-programmers as well
because they directly correspond to business processes.
I don't want to enumerate over all the advantages and disadvantages of
central coordination vs. federated collaboration here but central
coordination may be in conflict with availability requirements. Also,
central coordination is often the entry point to monolithic solutions :-)
Compare this to plain at-least-once based messaging between
persistent actors or services (in different datacenters, for example)
where you cannot make assumptions on message ordering and duplicates.
It usually makes writing correct business logic much harder.
Furthermore, if you want to decouple a service from the availability
of others you are again faced with the problem of detecting and
resolving conflicts. I should mention here that rejecting commands !=
being available :-). As soon as distributed services/applications
shall become more resilient to network partitions, you'll anyway have
to deal with many of the issues that Eventuate is already adressing.
Eventuate meanwhile emerged from a proof-of-concept in early 2015 to
a production ready toolkit for building (globally) distributed,
service-oriented CQRS/ES applications. As Eventuate is also almost a
functional superset of akka-persistence, I wonder if combining
efforts would make sense. Please let me know if this sounds
interesting to you.
I am very much interested in attacking this problem, we need to give
users the tools that reduce their responsibility to the essence of
their business problem. I must confess that I have not yet studied the
Eventuate source code, all statements above are based on my incomplete
understanding of the problem domain and some very interesting
conversations at ECOOP last year. On this basis my current impression
is that while we hold some promising pieces in our hands we have not
yet found the golden hammer—if it exists at all. Perhaps it is time to
plan a get-together to assemble the pieces we know now and explore the
landscape that emerges?
Good idea. Let's start that initiative.
Regards,
Roland
Regards,
Martin
So, you can use Akka Persistence with the same store in different
locations, but you’ll have to make sure that you don’t emit events
to the same log from different places—there can only be one running
source of truth for each persistenceId at any given time.
Regards,
Roland
22 jan 2016 kl. 14:52 skrev Paul Cleary <[email protected]>:
Will Akka Persistence work if you have two different clusters
pointing to the same data store?
Imagine I have 2 data centers that point at the same database.
If I have updates happening in both data centers at the same time,
will akka persistence stomp all over the journal / snapshots?
I know that akka persistence has sequence numbers, and I am not
sure how those are managed.
It would be great if this just worked, but I am thinking I need to
implement my own persistence plugin and / or persistence layer in
my app to make sure that there are no collisions.
--
>>>>>>>>>> Read the docs:http://akka.io/docs/
>>>>>>>>>> Check the
FAQ:http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the
archives:https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email [email protected]
<mailto:[email protected]>.
To post to this group, send email [email protected].
Visit this group athttps://groups.google.com/group/akka-user.
For more options, visithttps://groups.google.com/d/optout.
*Dr. Roland Kuhn*
/Akka Tech Lead/
Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
twitter: @rolandkuhn
<http://twitter.com/#%21/rolandkuhn>
--
>>>>>>>>>> Read the docs:http://akka.io/docs/
>>>>>>>>>> Check the
FAQ:http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives:https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email [email protected]
<mailto:[email protected]>.
To post to this group, send email [email protected]
<mailto:[email protected]>.
Visit this group athttps://groups.google.com/group/akka-user.
For more options, visithttps://groups.google.com/d/optout.
--
Martin Krasser
blog:http://krasserm.github.io
code:http://github.com/krasserm
twitter:http://twitter.com/mrt1nz
--
>>>>>>>>>> Read the docs:http://akka.io/docs/
>>>>>>>>>> Check the
FAQ:http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives:https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email [email protected]
<mailto:[email protected]>.
To post to this group, send email [email protected]
<mailto:[email protected]>.
Visit this group athttps://groups.google.com/group/akka-user.
For more options, visithttps://groups.google.com/d/optout.
*Dr. Roland Kuhn*
/Akka Tech Lead/
Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
twitter: @rolandkuhn
<http://twitter.com/#%21/rolandkuhn>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog: http://krasserm.github.io
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
