[
https://issues.apache.org/jira/browse/HDDS-17423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17423:
------------------------------
Attachment: TestBugMC1BootstrappedScmInPrefinalizedRing.java
> SCM bootstrapped into a ring that is not finalized starts finalized,
> finalizes datanodes without an admin finalize and gets them fenced
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17423
> URL: https://issues.apache.org/jira/browse/HDDS-17423
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0,
> 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-bootstrapped-scm-adopts-ring-apparent-version.patch,
> TestBugMC1BootstrappedScmInPrefinalizedRing.java
>
>
> h3. Mechanism
> {{StorageContainerManager.scmBootstrap}} takes only the cluster ID and the
> primary SCM ID from the running ring and writes a new VERSION file through
> {{new SCMStorageConfig(conf)}}, whose apparent version is
> {{HDDSVersion.SOFTWARE_VERSION}} unless the testing key
> {{testing.hdds.scm.init.apparent.version}} is set. {{ScmVersionManager}} is
> built from the VERSION file only, and the snapshot path
> ({{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}) and the
> FINALIZE apply can only raise the version. Nothing on the bootstrap or start
> path brings the new SCM down to the version of the other SCMs, so an SCM
> added while the ring is upgraded but not finalized runs finalized
> ({{needsFinalization() == false}}) next to SCMs that are not.
> h3. Trigger
> # All SCMs and datanodes run the new software and are not finalized yet.
> Today this is the state of a ring upgraded from 2.2.x to this code (apparent
> version is the legacy layout version); with zero downtime upgrade it is the
> state between the rolling upgrade of the binaries and the admin finalize.
> # An SCM is added with {{ozone scm --bootstrap}}, with production
> configuration.
> # The datanodes are configured with the new SCM, and the new SCM becomes
> leader (election or {{ozone admin scm transfer}}).
> h3. Impact
> * As leader, the new SCM sends {{FinalizeVersionCommand}} to every datanode
> ({{SCMNodeManager.sendFinalizeToDatanodeIfNeeded}}). The datanodes finalize
> although no admin finalize was issued and the other SCMs are still not
> finalized, so the downgrade option is lost (run).
> * A finalized datanode that restarts is rejected by each SCM that is not
> finalized ({{SCMNodeManager.shouldFenceDatanode}}, apparent version above
> SCM) and shuts down (run, three rejections, datanode stopped).
> * For an upgrade from a release that already has the zero downtime upgrade
> versions, the new SCM also rejects every datanode still on the previous
> release, which the other SCMs accept (run by sending the registration request
> of such a datanode to the SCM datanode protocol server, since no such release
> exists yet).
> * This lasts until an admin finalize. No data loss is expected.
> * Releases 1.3.0 to 2.2.1 have the same root cause: {{scmBootstrap}}
> initializes the new SCM at the newest layout version, so as leader it sends
> {{FinalizeNewLayoutVersionCommand}} to the datanodes of a cluster that is not
> finalized. Datanodes are not fenced there, because registration checks only
> the software layout version (read from code at the release tags, not run).
> h3. Reproduction
> PASS on the unmodified commit, about 64 s.
> [^TestBugMC1BootstrappedScmInPrefinalizedRing.java]
> ({{hadoop-ozone/integration-test}}) starts three SCMs and three datanodes at
> apparent version {{ZDU}} through the testing key, bootstraps a fourth SCM
> with {{MiniOzoneHAClusterImpl.bootstrapSCM}} without that key, adds it to the
> datanodes by reconfiguration and transfers leadership to it. Output on the
> unmodified code: the new SCM is at {{STREAM_PUTBLOCK_WITHOUT_RAFT (101)}}
> while the original SCMs are at {{ZDU (100)}}, all datanodes finalize to 101
> with no finalize command, a restarted datanode logs "SCM rejected this
> datanode's registration" and stops, and a registration with software and
> apparent version 100 is accepted by an original SCM and rejected by the new
> one. A passing test means the defect is present.
> h3. Patch
> [^MC-1-bootstrapped-scm-adopts-ring-apparent-version.patch], against
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
> e03ba37c41d. It targets master only; the release branches use the older
> layout version framework, where the same idea would need a separate change.
> {{GetScmInfoResponseProto}}, which {{scmBootstrap}} already fetches from the
> leader, gets an optional {{apparentVersion}} field filled by
> {{SCMBlockProtocolServer.getScmInfo}}. When it is lower than the initial
> apparent version of the new VERSION file, {{scmBootstrap}} writes it instead.
> A value that is too low is safe, because the new SCM finalizes with the ring
> when it applies the FINALIZE log entry (run) or installs a snapshot of a
> finalized ring (from code). A finalized ring and an existing SCM directory
> are unchanged. The field is only in a read RPC response and not in the Raft
> log; a response without it (a leader without the patch) keeps today's
> behaviour. A ring version above this SCM's software version is not used and
> is logged as a warning.
> Not covered: the FINALIZE log entry has no target version, so a FINALIZE
> entry from an earlier upgrade that is still in the retained Raft log would
> raise a replaying new SCM to its own software version. That cannot happen at
> this commit: the entry is written only by this code, and a ring is
> unfinalized here only during its first upgrade into this framework, from a
> release whose log has no such entry.
> The existing {{queryUpgradeStatus}} RPC already returns the SCM apparent
> version, but it fails while SCM is in safe mode, so using it would make
> bootstrap fail in a case it supports today.
> Covered by {{testBootstrappedScmJoinsRingAtRingApparentVersion}} in the
> existing {{TestScmHAFinalization}}: a ring at {{INITIAL_VERSION}} bootstraps
> an SCM without the testing key, which must join at {{INITIAL_VERSION}} and
> then finalize with the ring; a second SCM bootstrapped into the finalized
> ring must start at the software version. {{testGetScmInfoApparentVersion}} in
> the existing {{TestScmBlockLocationProtocolPB}} checks that a response
> without the field gives no version, so bootstrap keeps today's behaviour.
> Without the change it fails with {{expected: INITIAL_VERSION (0) but was:
> STREAM_PUTBLOCK_WITHOUT_RAFT (101)}}. With it, {{TestScmHAFinalization}},
> {{TestStorageContainerManagerHA}}, {{TestDatanodeSCMNodesReconfiguration}},
> {{TestFailoverWithSCMHA}}, {{TestMiniOzoneHACluster}},
> {{TestSecureOzoneCluster}} and the SCM unit suites for the block protocol
> server, both block protocol translators, the version manager and SCM startup
> pass, and checkstyle is clean. With the patch the reproduction fails at its
> first assertion on the new SCM, because it starts at {{ZDU (100)}}.
> Related: HDDS-17089 (an SCM restarted after an interrupted snapshot install
> stays behind its finalized DB) and HDDS-17090 (a lagging follower rejects
> datanodes the leader finalized) are different mechanisms and touch different
> code; the patches do not overlap.
> Found by TLA+ model checking and code review of the zero downtime upgrade
> version gating in SCM, OM and the datanode write path under HDDS-15926, on
> commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues
> and apache/ozone pull requests for duplicates before filing. The attached
> patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
