[
https://issues.apache.org/jira/browse/ARTEMIS-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Nigro updated ARTEMIS-3340:
-------------------------------------
Summary: Replicated Journal quorum-based logical timestamp/version (was:
Replicated Journal quorum-based logical timestamp)
> Replicated Journal quorum-based logical timestamp/version
> ---------------------------------------------------------
>
> Key: ARTEMIS-3340
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3340
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Reporter: Francesco Nigro
> Priority: Major
>
> Shared-nothing replication can cause journal misalignment despite no
> split-brain events.
> There are several ways that can cause this to happen.
> Below some scenario that won't involve network partitions/drastic outages.
> Scenario 1:
> # Master/Primary start as live, clients connect to it
> # Backup become an in-sync replica
> # User stop live and backup failover to it
> # *Backup serve clients alone, modifying its journal*
> # User stop backup
> # User start master/primary: it become live with a journal misaligned to the
> most up-to-date one ie on the stopped backup
> Scenario 2:
> # Master/Primary start as live, clients connect to it
> # Backup become an in-sync replica
> # Connection glitch between backup -> live
> # backup start trying to failover (for {{vote-retries * vote-retry-wait}}
> milliseconds)
> # *Live serve clients alone, modifying its journal*
> # User stop live
> # Backup succeed to failover: it become live with a journal misaligned to
> the most up-to-date one ie on the stopped live
> The main cause of this issue is because we allow *a single broker to serve
> clients*, despite configured with HA, generating the journal misalignment.
> The quorum service (classic or pluggable) just take care of mutual exclusive
> presence of broker for the live role (vs a NodeID), without considering live
> role ordering based on the most up-to-date journal.
> A possible solution is to use
> https://issues.apache.org/jira/browse/ARTEMIS-2716 and use a quorum "logical
> timestamp" marking the age of the journal in order to force live to always
> have the most up-to-date journal. It means that the same
> In case of quorum service restart/outage, admin must use
> command/configuration to let a broker to ignore the age of its journal and
> just force it to start.
> In addition new journal CLI commands should be implemented to inspect the age
> of a (local) broker journal or query/force the quorum journal version, for
> troubleshooting reasons.
> It's very important to capture every possible event that cause the journal
> age to increase
> eg
> # live broker send its journal file to a not yet in sync replica backup,
> along with its "journal age"
> # backup is now ready to failover in any moment
> # a network partition happen
> # backup try to become live for vote-retries times
> # live detect replication disconnection but is "lucky" that can reach the
> quorum and continue serving clients
> # live increment the age of its journal
> # an outage cause live to die
> # network partition is restored
> # backup detect that journal age is no longer matching its own journal: it
> stop trying to become live
> The key parts related to journal age/version are:
> * only who's live can change journal version (with a monotonic increment)
> * every breaking point event must cause journal age/version to change eg
> starting as live, loosing its backup, etc etc
>
> Re the RI implementation using Apache Curator, this could use a separate
> [DistributedAtomicLong|https://curator.apache.org/apidocs/org/apache/curator/framework/recipes/atomic/DistributedAtomicLong.html]
> to manage the journal version.
> Although tempting, it's not a good idea to use the data field on
> {{InterProcessSemaphoreV2}}, because:
> * there's no API to query it if no lease is acquired yet (or created)
> * we more need to "age" the journal independently from the lock
> acquisition/release process eg a live that drop its replica need to increment
> the journal version
> Athough tempting, it's not a good idea to just use the last alive broker
> connector identity instead of a journal version, because of the ABA problem
> (see https://en.wikipedia.org/wiki/ABA_problem).
> This versioning mechanism isn't without drawbacks: quorum journal versioning
> requires to store a local copy of the version in order to allow the broker to
> query and compare it with the quorum one on restart; having 2 separate and
> not atomic operations means that there must be a way to reconcile/fix it in
> case of misalignments.
> This could be done with admin operations.
> The versioning change the way roles behave, but they still retain theirs key
> characteristics:
> - backup can start as live in case of most up to date journal and no other
> live around, but if not, can just rotate journal and be available to sync
> with a live
> - primary try to failback to any existing live with the most up to date
> journal or await it, without becoming live in case of old journal
> This would ensure that If both broker are up and running and backup allow
> primary to failback, the primary eventually become live and backup replicates
> it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
