[
https://issues.apache.org/jira/browse/ARTEMIS-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17360699#comment-17360699
]
Gary Tully edited comment on ARTEMIS-3340 at 6/28/21, 11:16 AM:
----------------------------------------------------------------
it may be worth considering a more restrictive (and smaller/simpler) set of
criteria.
Disallow a transition from REPLICATED to UNREPLICATED with the same lock
version. ie: a LIVE that is REPLICATED (has an insync replica/backup), must
exit or release the Journal lock if it is no longer REPLICATED. Ie: a failover
event must occur and the lock version must increment.
To proceed, either the insync-replica gets the lock with a matching version and
increments, or the restarted live gets the lock with matching version and
increments. That is, both behave the same. If they get the lock and cannot
increment the version they exit/release.
This means that if the most uptodate journal is not available we are
unavailable. But we never have a journal out of sync.
For the single pair scenario, if there is a need for HA to survive two
failures, introduce a second pair.
This greatly simplifies logic and state. let me try and reason via text ... (I
think that is the test of whether it is understandable)
The lock has an auto increment version, journal updates only occur with the
lock and record the version. Imagine every journal record having an additional
long field.
an empty journal needs an empty lock. The first journal append will have ver=1.
Live replicates to Backup that stores ver=1 in its journal. The next
incarnation of the journal (either restarted live or backup) will use ver=2 etc.
it begs the question, we get the lock at ver=1 and increment to ver=2 but we
fail to record any journal record with ver=2 -> we are unavailable!
we need to rollback the lock version to proceed. that is the downside of having
to coordinate but it is the only way to be safe.
was (Author: gtully):
it may be worth considering a more restrictive (and smaller/simpler) set of
criteria.
Disallow a transition from REPLICATED to UNREPLICATED with the same lock
version. ie: a LIVE that is REPLICATED (has an insync replica/backup), must
exit or release the Journal lock if it is no longer REPLICATED. Ie: a failover
event must occur and the lock version must increment.
To proceed, either the insync-replica gets the lock with a matching version and
increments, or the restarted live gets the lock with matching version and
increments. That is, both behave the same. If they get the lock and cannot
increment the version they exit/release.
This means that if the most uptodate journal is not available we are
unavailable. But we never have a journal out of sync.
For the single pair scenario, if there is a need for HA to survive two
failures, introduce a second pair.
This greatly simplifies logic and state. let me try and reason via text ... (I
think that is the test of whether it is understandable)
The lock has an auto increment version, journal updates only occur with the
lock and record the version. Imagine every journal record having an additional
long field.
an empty journal needs an empty lock. The first journal append will have ver=1.
Live replicates to Backup that stores ver=1 in its journal. The next
incarnation of the journal (either restarted live or backup) will use ver=2 etc.
it begs the question, we get the lock at ver=1 and increment to ver=2 but we
fail to record any journal record with ver=2 -> we are unavailable!
we need to rollback the lock version to proceed. that is the downside of having
to coordinate but it is the only way to be safe.
is there any way to be smarter here?
> Replicated Journal quorum-based logical timestamp/version
> ---------------------------------------------------------
>
> Key: ARTEMIS-3340
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3340
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Reporter: Francesco Nigro
> Priority: Major
>
> Shared-nothing replication can cause journal misalignment despite no
> split-brain events.
> There are several ways that can cause this to happen.
> Below some scenario that won't involve network partitions/drastic outages.
> Scenario 1:
> # Master/Primary start as live, clients connect to it
> # Backup become an in-sync replica
> # User stop live and backup failover to it
> # *Backup serve clients alone, modifying its journal*
> # User stop backup
> # User start master/primary: it become live with a journal misaligned to the
> most up-to-date one ie on the stopped backup
> Scenario 2:
> # Master/Primary start as live, clients connect to it
> # Backup become an in-sync replica
> # Connection glitch between backup -> live
> # backup start trying to failover (for {{vote-retries * vote-retry-wait}}
> milliseconds)
> # *Live serve clients alone, modifying its journal*
> # User stop live
> # Backup succeed to failover: it become live with a journal misaligned to
> the most up-to-date one ie on the stopped live
> The main cause of this issue is because we allow *a single broker to serve
> clients*, despite configured with HA, generating the journal misalignment.
> The quorum service (classic or pluggable) just take care of mutual exclusive
> presence of broker for the live role (vs a NodeID), without considering live
> role ordering based on the most up-to-date journal.
> A possible solution is to use
> https://issues.apache.org/jira/browse/ARTEMIS-2716 and use a quorum "logical
> timestamp/version" marking the age/ownership changes of the journal in order
> to force live to always have the most up-to-date journal. It means that such
> value has to be locally saved and exchanged during the initial replica sync,
> involving both journal data and core message protocol change (valid just for
> the replication channel, without impacting clients).
> In case of quorum service restart/outage, admin must use
> command/configuration to let a broker to ignore the age of its journal and
> just force it to start.
> In addition new journal CLI commands should be implemented to inspect the age
> of a (local) broker journal or query/force the quorum journal version too,
> for troubleshooting reasons.
> It's very important to capture every possible event that cause the journal
> age/ownership to change
> eg Possible scenario 2 (again):
> # live broker start because it matches the most up to date journal version,
> increasing it (locally and remotely) when it become fully alive
> # backup found it and trust that, given that's live, it already has the
> most-up-to-date journal for a specific NodeID
> # live broker send its journal files to the backup, along with its local
> journal version
> # backup is now ready to failover in any moment
> # network glitch happen
> # backup try to become live for vote-retries times
> # live detect replication disconnection and *increment the journal version*
> (quorum and locally)
> # live serve clients alone, modifying its journal
> # an outage/stop cause live to die
> # backup detect that *journal version no longer matching its own local
> journal version*: it stop trying to become live
> The key parts related to journal age/version are:
> * only who's live can change quorum (and local) journal version (with a
> monotonic increment)
> * every ownership change event must cause journal age/version to change eg
> starting as live, loosing its backup, etc etc
> Re the RI implementation using Apache Curator, this could use a separate
> [DistributedAtomicLong|https://curator.apache.org/apidocs/org/apache/curator/framework/recipes/atomic/DistributedAtomicLong.html]
> to manage the journal version.
> Although tempting, it's not a good idea to not just use the data field on
> {{InterProcessSemaphoreV2}}, because:
> * there's no API to query it if no lease is acquired yet (or created)
> * we more need to "age" the journal independently from the lock
> acquisition/release process eg a live that drop its replica need to increment
> the journal version
> Athough tempting, it's not a good idea to just use the last alive broker
> connector identity instead of a journal version, because of the ABA problem
> (see https://en.wikipedia.org/wiki/ABA_problem).
> This versioning mechanism isn't without drawbacks: quorum journal versioning
> requires to store a local copy of the version in order to allow the broker to
> query and compare it with the quorum one on restart; having 2 separate and
> not atomic operations means that there must be a way to reconcile/fix it in
> case of misalignments. As said above, this could be done with admin
> operations.
> Journal versioning change the way roles behave, but they still retain theirs
> key characteristics:
> - backup should try start as live in case it has the most up to date journal
> and there is no other live around: differently, can just rotate journal and
> be available to replicate some live
> - primary try to fail-back to any existing live with the most up to date
> journal or await it to appear, without becoming live if it doesn't have the
> most up-to-date journal
> This would ensure that If both broker are up and running and backup allow a
> primary to failback, the primary eventually become live and backup replicates
> it, preserving the desired broker roles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
