Siyao Meng created HDDS-17421:
---------------------------------
Summary: SCM terminates when it applies a Raft log entry written
by the pre-ZDU SCM finalization, and again on every restart
Key: HDDS-17421
URL: https://issues.apache.org/jira/browse/HDDS-17421
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-9-apply-pre-zdu-scm-finalization-log-entries.patch,
TestBugCR9PreZduFinalizeEntryReplay.java
h3. Mechanism
When SCM finalizes on releases 1.3.0 to 2.2.1, the leader writes FINALIZE Raft
log entries named {{addFinalizingMark}}, {{finalizeLayoutFeature}} (one
{{Integer}} argument, once per layout feature) and {{removeFinalizingMark}}. On
master the generated {{FinalizationStateManagerInvoker.invokeLocal}} only knows
{{finalizeUpgrade}} and {{reinitialize}} (HDDS-14570, HDDS-14672, HDDS-15377),
so applying one of the old entries throws {{IllegalArgumentException("Method
not found: ... in FinalizationStateManager")}}. That is not an
{{SCMException}}, so {{SCMStateMachine.applyTransaction}} calls
{{ExitUtils.terminate}}. The entry is still unapplied, so the SCM replays it
and terminates again on every start. The request encoding
({{SCMRatisProtocol.proto}}, {{SCMRatisRequest}}, codecs) is unchanged since
2.2.1, only the method name lookup fails. HDDS-16169 fixed the same failure for
the renamed {{ContainerStateManager.updateContainerState}}.
h3. Trigger
An SCM has to apply such an entry on the new code, which happens when it did
not apply it before the upgrade:
# An SCM that was down or behind while its ring finalized an upgrade on the old
release, and is then started on the new release. It catches up by log append:
with the default {{ozone.scm.ha.ratis.log.purge.enabled}} = false the leader
does not purge entries that another SCM has not committed yet.
# An SCM stopped uncleanly (crash, kill) after the old finalization and before
a transaction buffer flush or snapshot covered it, then started on the new
release. A graceful stop takes a snapshot first and is not affected. HDDS-9769
is the same replay with the old code.
A freshly bootstrapped SCM is normally not affected: it installs the leader's
checkpoint before it applies any entry (seen in
{{TestStorageContainerManagerHA#testBootStrapSCM}} with an unpurged log). It
replays the whole log only in the fallback of HDDS-16899.
h3. Impact
The affected SCM cannot start on the new release. If all SCMs were stopped
uncleanly right after the old finalization, no SCM can start. No data loss.
Master only, no release has it. A workaround (argued, not run) is to start the
stuck SCM on the old release until it has applied the entries, then upgrade it
again; it was never finalized on the new release, so the old release can still
run it.
h3. Reproduction
PASS on unmodified source. [^TestBugCR9PreZduFinalizeEntryReplay.java]
({{hadoop-hdds/server-scm}}) starts one real SCM with its own single node Ratis
ring at the pre-ZDU layout version {{HBASE_SUPPORT}}, appends a
{{finalizeLayoutFeature(9)}} entry encoded as 2.2.1 encodes it through the
SCM's Ratis server, and restarts the SCM once. The SCM terminates with {{Method
not found: finalizeLayoutFeature in FinalizationStateManager}} on apply and
again after the restart. A passing test means the defect is present. Not run
against a real 2.2.x ring.
h3. Patch
[^CR-9-apply-pre-zdu-scm-finalization-log-entries.patch], against
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
e03ba37c41d.
As in HDDS-16169, {{FinalizationStateManager}} declares the three old methods
again with {{@Deprecated}}, and {{FinalizationStateManagerInvoker}} is
regenerated. Unlike HDDS-16169 they are not {{@Replicate}}, so this version
cannot write them while {{invokeLocal}} still dispatches them.
{{addFinalizingMark}} and {{removeFinalizingMark}} do nothing (the mark is
unused, {{ClearFinalizingStateScmUpgradeAction}} removes a leftover one).
{{finalizeLayoutFeature}} finalizes up to the given layout version and no
further, as the SCM that wrote it did, using a new
{{ComponentVersionManager.finalizeUpgrade(ComponentVersion)}} that stops at the
target; {{finalizeUpgrade()}} delegates to it with the software version and
behaves as before. A no-op would leave that SCM below its finalized datanodes,
which it then fences (as in HDDS-17090); finalizing to the software version
would finalize it past ZDU without an admin finalize. No Raft log or proto
change.
Covered by {{testPreZduFinalizationEntriesFinalizeUpToTheirLayoutVersion}} in
the existing {{TestSCMStateMachine}}: the old entries applied to a real
{{ScmVersionManager}} at {{HBASE_SUPPORT}} through the real invoker end at
{{STORAGE_SPACE_DISTRIBUTION}}, a replayed entry changes nothing, a new
{{finalizeUpgrade}} entry still finalizes to the software version, and an
unknown operation or an unknown layout version still terminates. Without the
change it fails with {{Method not found: addFinalizingMark in
FinalizationStateManager}}. With it, {{TestSCMStateMachine}},
{{TestScmInvokerCodeGenerator}}, the SCM, datanode and OM version manager
tests, the SCM upgrade action and startup tests, {{TestSCMHAManagerImpl}} and
{{TestScmHAFinalization}} pass, and checkstyle is clean. With the patch the
reproduction fails at its first assertion, because the entry is applied and the
SCM finalizes to {{WITNESSED_CONTAINER_DB_PROTO_VALUE}}.
Found by code review of the zero downtime upgrade version gating in SCM, OM and
the datanode write path, as part of the TLA+ verification effort under
HDDS-15926, on commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against
HDDS issues and apache/ozone pull requests for duplicates before filing. The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
