Siyao Meng created HDDS-17423:
---------------------------------
Summary: SCM bootstrapped into a ring that is not finalized starts
finalized, finalizes datanodes without an admin finalize and gets them fenced
Key: HDDS-17423
URL: https://issues.apache.org/jira/browse/HDDS-17423
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-1-bootstrapped-scm-adopts-ring-apparent-version.patch,
TestBugMC1BootstrappedScmInPrefinalizedRing.java
h3. Mechanism
{{StorageContainerManager.scmBootstrap}} takes only the cluster ID and the
primary SCM ID from the running ring and writes a new VERSION file through
{{new SCMStorageConfig(conf)}}, whose apparent version is
{{HDDSVersion.SOFTWARE_VERSION}} unless the testing key
{{testing.hdds.scm.init.apparent.version}} is set. {{ScmVersionManager}} is
built from the VERSION file only, and the snapshot path
({{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}) and the FINALIZE
apply can only raise the version. Nothing on the bootstrap or start path brings
the new SCM down to the version of the other SCMs, so an SCM added while the
ring is upgraded but not finalized runs finalized ({{needsFinalization() ==
false}}) next to SCMs that are not.
h3. Trigger
# All SCMs and datanodes run the new software and are not finalized yet. Today
this is the state of a ring upgraded from 2.2.x to this code (apparent version
is the legacy layout version); with zero downtime upgrade it is the state
between the rolling upgrade of the binaries and the admin finalize.
# An SCM is added with {{ozone scm --bootstrap}}, with production configuration.
# The datanodes are configured with the new SCM, and the new SCM becomes leader
(election or {{ozone admin scm transfer}}).
h3. Impact
* As leader, the new SCM sends {{FinalizeVersionCommand}} to every datanode
({{SCMNodeManager.sendFinalizeToDatanodeIfNeeded}}). The datanodes finalize
although no admin finalize was issued and the other SCMs are still not
finalized, so the downgrade option is lost (run).
* A finalized datanode that restarts is rejected by each SCM that is not
finalized ({{SCMNodeManager.shouldFenceDatanode}}, apparent version above SCM)
and shuts down (run, three rejections, datanode stopped).
* For an upgrade from a release that already has the zero downtime upgrade
versions, the new SCM also rejects every datanode still on the previous
release, which the other SCMs accept (run by sending the registration request
of such a datanode to the SCM datanode protocol server, since no such release
exists yet).
* This lasts until an admin finalize. No data loss is expected.
* Releases 1.3.0 to 2.2.1 have the same root cause: {{scmBootstrap}}
initializes the new SCM at the newest layout version, so as leader it sends
{{FinalizeNewLayoutVersionCommand}} to the datanodes of a cluster that is not
finalized. Datanodes are not fenced there, because registration checks only the
software layout version (read from code at the release tags, not run).
h3. Reproduction
PASS on the unmodified commit, about 64 s.
[^TestBugMC1BootstrappedScmInPrefinalizedRing.java]
({{hadoop-ozone/integration-test}}) starts three SCMs and three datanodes at
apparent version {{ZDU}} through the testing key, bootstraps a fourth SCM with
{{MiniOzoneHAClusterImpl.bootstrapSCM}} without that key, adds it to the
datanodes by reconfiguration and transfers leadership to it. Output on the
unmodified code: the new SCM is at {{STREAM_PUTBLOCK_WITHOUT_RAFT (101)}} while
the original SCMs are at {{ZDU (100)}}, all datanodes finalize to 101 with no
finalize command, a restarted datanode logs "SCM rejected this datanode's
registration" and stops, and a registration with software and apparent version
100 is accepted by an original SCM and rejected by the new one. A passing test
means the defect is present.
h3. Patch
[^MC-1-bootstrapped-scm-adopts-ring-apparent-version.patch], against
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
e03ba37c41d. It targets master only; the release branches use the older layout
version framework, where the same idea would need a separate change.
{{GetScmInfoResponseProto}}, which {{scmBootstrap}} already fetches from the
leader, gets an optional {{apparentVersion}} field filled by
{{SCMBlockProtocolServer.getScmInfo}}. When it is lower than the initial
apparent version of the new VERSION file, {{scmBootstrap}} writes it instead. A
value that is too low is safe, because the new SCM finalizes with the ring when
it applies the FINALIZE log entry (run) or installs a snapshot of a finalized
ring (from code). A finalized ring and an existing SCM directory are unchanged.
The field is only in a read RPC response and not in the Raft log; a response
without it (a leader without the patch) keeps today's behaviour. A ring version
above this SCM's software version is not used and is logged as a warning.
Not covered: the FINALIZE log entry has no target version, so a FINALIZE entry
from an earlier upgrade that is still in the retained Raft log would raise a
replaying new SCM to its own software version. That cannot happen at this
commit: the entry is written only by this code, and a ring is unfinalized here
only during its first upgrade into this framework, from a release whose log has
no such entry.
The existing {{queryUpgradeStatus}} RPC already returns the SCM apparent
version, but it fails while SCM is in safe mode, so using it would make
bootstrap fail in a case it supports today.
Covered by {{testBootstrappedScmJoinsRingAtRingApparentVersion}} in the
existing {{TestScmHAFinalization}}: a ring at {{INITIAL_VERSION}} bootstraps an
SCM without the testing key, which must join at {{INITIAL_VERSION}} and then
finalize with the ring; a second SCM bootstrapped into the finalized ring must
start at the software version. {{testGetScmInfoApparentVersion}} in the
existing {{TestScmBlockLocationProtocolPB}} checks that a response without the
field gives no version, so bootstrap keeps today's behaviour. Without the
change it fails with {{expected: INITIAL_VERSION (0) but was:
STREAM_PUTBLOCK_WITHOUT_RAFT (101)}}. With it, {{TestScmHAFinalization}},
{{TestStorageContainerManagerHA}}, {{TestDatanodeSCMNodesReconfiguration}},
{{TestFailoverWithSCMHA}}, {{TestMiniOzoneHACluster}},
{{TestSecureOzoneCluster}} and the SCM unit suites for the block protocol
server, both block protocol translators, the version manager and SCM startup
pass, and checkstyle is clean. With the patch the reproduction fails at its
first assertion on the new SCM, because it starts at {{ZDU (100)}}.
Related: HDDS-17089 (an SCM restarted after an interrupted snapshot install
stays behind its finalized DB) and HDDS-17090 (a lagging follower rejects
datanodes the leader finalized) are different mechanisms and touch different
code; the patches do not overlap.
Found by TLA+ model checking and code review of the zero downtime upgrade
version gating in SCM, OM and the datanode write path under HDDS-15926, on
commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues
and apache/ozone pull requests for duplicates before filing. The attached patch
is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
