Siyao Meng created HDDS-17016:
---------------------------------
Summary: Datanode with more failed volumes than tolerated keeps
serving after restart, then stops on a later volume check that finds nothing
wrong
Key: HDDS-17016
URL: https://issues.apache.org/jira/browse/HDDS-17016
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-12-enforce-volume-failure-tolerance-on-every-check.patch,
TestBugMC12ToleranceAtStartup.java
h3. Mechanism
{{MutableVolumeSet.hasEnoughVolumes}} is only called from
{{handleVolumeFailures}}, and {{checkAllVolumes}} calls that only when the
check itself finds a new failure. Volumes that are failed directly through
{{failVolume}} are never compared with
{{hdds.datanode.failed.data.volumes.tolerated}} (or its metadata and DB
variants) at that point. At startup these are volumes rejected by
{{initializeVolumeSet}} (inconsistent or unloadable root), by
{{VersionEndpointTask.checkVolumeSet}} and by {{ContainerReader}}; at runtime
{{ScmHAFinalizeUpgradeActionDatanode}} does the same. The startup
{{checkAllVolumes}} in {{OzoneContainer.start}} then finds no new failure, so
the datanode starts and serves although it is over the limit.
The callback of {{checkVolumeAsync}} calls {{handleVolumeFailures}} even when
the check found nothing. The next on demand check of any healthy volume
(started by an I/O error on a container) therefore finds the set over the limit
and calls {{handleFatalVolumeFailures}}, stopping the datanode at an arbitrary
later time.
This dates from the introduction of the tolerance: the startup exception that
HDDS-9022 replaced with {{handleFatalVolumeFailures}} was also only thrown from
{{handleVolumeFailures}}.
h3. Trigger
A finite tolerance and a restart with more unusable volumes than tolerated, for
example volumes whose {{VERSION}} file is missing or corrupt. With the default
{{-1}} only one corner case is affected: {{ContainerReader}} at startup or
{{ScmHAFinalizeUpgradeActionDatanode}} at runtime fails every data volume
directly. The datanode then keeps running with no data volume, because no
healthy volume is left to start an on demand check.
h3. Impact
* The datanode serves with more failed volumes than the operator allowed, until
any later failure handling runs.
* It is then stopped by a check that found nothing wrong, for example the on
demand check of a healthy volume.
* Shown at unit level with a real volume set and checker and a mocked
{{DatanodeStateMachine}}; not exercised on a datanode registered with SCM.
h3. Reproduction
PASS on unmodified source. [^TestBugMC12ToleranceAtStartup.java]
({{hadoop-hdds/container-service}}, target path and Maven command in the class
comment) sets the data volume tolerance to 1, formats three volumes through
{{StorageVolumeUtil.checkVolume}}, removes the {{VERSION}} file of two,
rebuilds the volume set with a real {{StorageVolumeChecker}} and runs
{{checkAllVolumes()}} as {{OzoneContainer.start}} does. Output: {{healthy=1
failed=2 tolerated=1}}, and the startup check makes no fatal call.
{{StorageVolumeUtil.onFailure}} on the healthy volume then logs "Not enough
volumes ... MaxVolumeFailuresTolerated: 1, ActiveVolumes: 1, FailedVolumes: 2"
and calls {{handleFatalVolumeFailures}} while the checked volume is still
healthy. A passing test means the defect is present.
h3. Patch
[^MC-12-enforce-volume-failure-tolerance-on-every-check.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
{{MutableVolumeSet.checkAllVolumes}} now also calls
{{handleFatalVolumeFailures}} when the check found no new failure but
{{hasEnoughVolumes()}} is false. {{OzoneContainer.start}} runs it on the data,
metadata and DB volume sets after all startup paths that fail volumes directly,
so an over limit datanode stops at startup, and the periodic check covers
direct failures at runtime. Stopping at startup is the intended behaviour:
HDDS-5219 threw "Don't have enough good volumes on startup", and
{{TestDatanodeHddsVolumeFailureToleration.testDNCorrectlyHandlesVolumeFailureOnStartup}}
expects a restart with two bad volumes and tolerance 1 to exit. Both only work
when the startup check itself finds the failures; the patch applies the same
rule to volumes that failed before that check. Calling {{handleVolumeFailures}}
unconditionally was not chosen because it would also run the failed volume
listener on every periodic check.
Covered by {{testVolumesFailedOnLoadCountTowardsTolerance}} in the existing
{{TestVolumeSetDiskChecks}}: three volumes, two of them failed by
{{initializeVolumeSet}}, a checker that reports no failure. With tolerance 1 it
expects one fatal call and without the change fails with "Wanted but not
invoked: datanodeStateMachine.handleFatalVolumeFailures()"; with tolerance 2
and with the default {{-1}} it expects none and passes either way. With the
change the volume, container, reader, upgrade, handler and state machine suites
(309 tests) and {{TestDatanodeHddsVolumeFailureToleration}} pass, and
checkstyle is clean. With the patch the reproduction fails at its check that
startup made no fatal call.
*Upgrade note.* A datanode with a finite
{{hdds.datanode.failed.data.volumes.tolerated}} (or the metadata or DB variant)
that has more unusable volumes than tolerated now exits at startup with
"DatanodeStateMachine Shutdown due to too many bad volumes", instead of serving
until a later volume check stops it. Such a datanode will not come back after
the upgrade restart, so a rolling upgrade stalls at it. Before upgrading,
compare {{NumFailedVolumes}} of {{VolumeHealthMetrics-DATA_VOLUME}} (and
{{VolumeHealthMetrics-META_VOLUME}}, {{VolumeHealthMetrics-DB_VOLUME}}) with
the tolerance on each such datanode, then repair or remove the failed
directories or raise the tolerance. With the default {{-1}} a datanode now also
exits when every data volume has been failed directly, instead of running with
none.
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-5219 (introduced the tolerance), HDDS-9022 (replaced the
startup exception with the fatal handler), HDDS-14871 (volume check latch
timeout) and HDDS-11770 (proposed default 0, abandoned). The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
