Siyao Meng created HDDS-16862:
---------------------------------
Summary: Finalization from a Ratis snapshot should stop at the
version recorded in the snapshot (OM, SCM)
Key: HDDS-16862
URL: https://issues.apache.org/jira/browse/HDDS-16862
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-bound-snapshot-finalization-to-snapshot-version.patch
h3. Mechanism
When an OM installs a Ratis snapshot, {{OzoneManager.instantiateServices}}
calls {{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}. SCM calls
the same method from {{FinalizationStateManagerImpl.reinitialize}}. If the
{{#LAYOUTVERSION}} in the installed DB is ahead of the local apparent version,
the method logs "Attempting to finalize to that version" and then calls
{{ComponentVersionManager.finalizeUpgrade()}}. That call does not stop at the
snapshot's version. It follows {{nextVersion()}} up to the local software
version, runs the upgrade action of every version on the way, and the method
then writes the local software version into the installed DB.
So whenever the snapshot records a version below the local software version,
the node finalizes further than the leader it took the snapshot from. The pre
ZDU code added by HDDS-4908 behaved the same way: it finalized to the full
software layout version and only checked afterwards that the result was not
below the DB version.
h3. Trigger
# An OM (or SCM) is down while its peers finalize to some version V.
# The cluster is then upgraded to a release whose software version is above V
and is not finalized yet. The down node is started on that release.
# It installs a snapshot from the leader whose DB records V. Instead of
stopping at V, it finalizes to its own software version.
This works with the current version catalogs. For OM, V can be any pre ZDU
layout feature, and the node then finalizes to {{ZDU (100)}}.
h3. Impact
Downgrade after finalization is not supported by design. The problem is that
one node ends up finalized further than its peers:
* Its VERSION file and {{#LAYOUTVERSION}} are ahead of its peers, and nothing
lowers them again. It has run the upgrade actions of versions its peers have
not finalized, for OM including the {{ZDU}} action.
* That OM alone reports the newer version to clients and allows the features
gated on it. While it is leader, those features are in use although the other
OMs have not finalized them (argued, not run).
* The newer release is not finalized on its peers, so the design still allows
downgrading it, but this node holds a version the older release does not know
and cannot start on it (argued, not run).
h3. Patch
[^MC-4-bound-snapshot-finalization-to-snapshot-version.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
{{ComponentVersionManager}} gets {{finalizeUpgrade(ComponentVersion
lastVersion)}}, which runs the existing loop and stops after {{lastVersion}}.
{{finalizeUpgrade()}} delegates with the software version, so its behaviour is
unchanged. {{finalizeFromSnapshotIfRequired}} passes the version read from the
snapshot DB, so OM and SCM finalize from a snapshot exactly to the version the
snapshot records, and run only the upgrade actions up to it. A snapshot from a
fully finalized leader of the same release records the software version, so
that case behaves as before.
Regression tests in the patch:
{{testFinalizeFromSnapshotStopsAtSnapshotVersion}} in the existing
{{TestOMVersionManager}} and {{TestScmVersionManager}}, using the real version
catalogs and VERSION files. Each test covers three cases:
* A snapshot between the node's apparent and software versions: the node stops
at the snapshot's version and stays pre-finalized. Its upgrade action runs and
the {{ZDU}} action does not.
* A snapshot at the software version: the node finalizes to it.
* A snapshot that is not ahead, or a DB without a version: nothing changes.
Without the change they fail with "expected: <HBASE_SUPPORT (7)> but was: <ZDU
(100)>" (OM) and "expected: <ERASURE_CODED_STORAGE_SUPPORT (3)> but was: <ZDU
(100)>" (SCM). With it, the following pass and checkstyle is clean:
* {{TestOMVersionManager}}, {{TestScmVersionManager}} and
{{TestDatanodeVersionManager}}.
* The OM finalize request and service unit tests.
* {{TestOMUpgradeFinalization}} and {{TestScmHAFinalization}} on a mini
cluster, including their snapshot finalization tests.
Found by TLA+ model checking and code review of the OM upgrade finalization
paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
