Siyao Meng created HDDS-17010:
---------------------------------
Summary: Datanode volume check waits the full disk check timeout
on an empty volume set, delaying the check of the next volume set
Key: HDDS-17010
URL: https://issues.apache.org/jira/browse/HDDS-17010
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-2-return-at-once-for-empty-volume-list.patch,
TestBugCR2EmptyVolumeSetCheck.java
h3. Mechanism
{{StorageVolumeChecker.checkAllVolumes}} creates a {{CountDownLatch}} of 1 and
counts it down only inside the loop over the volumes (in the result callback,
or when the last volume's check is skipped). For an empty collection the loop
does not run, so {{latch.await}} always waits the full
{{hdds.datanode.disk.check.timeout}} (10 minutes by default) and then logs
"checkAllVolumes timed out". {{checkAllVolumeSets}} is synchronized and runs
the sets one after the other in registration order (data, metadata, db) on one
thread, so every set after the empty one is checked that much later, in every
periodic cycle. The periodic check is scheduled with a fixed delay, so every
later cycle also starts that much later, even when the empty set is the last
one.
h3. Trigger
# {{hdds.datanode.failed.data.volumes.tolerated}} (or the metadata or db
variant) is set to at least the number of volumes in that set. The tolerance is
not capped at the number of configured volumes.
# Every volume in that set fails a check. {{hasEnoughVolumes}} is still true,
so the datanode keeps running with the empty set registered.
# Each later periodic check waits the timeout on the empty set.
An empty set cannot occur at startup ({{initializeVolumeSet}} fails) or with
the default tolerance of -1 (the datanode stops when the set becomes empty).
h3. Impact
* Failure detection for the volume sets registered after the empty one is
delayed by the disk check timeout in every cycle, and every later cycle starts
that much later. For metadata and db volumes the periodic check is the only
check (argued from the code).
* A misleading "checkAllVolumes timed out" warning is logged every cycle.
* No data impact.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugCR2EmptyVolumeSetCheck.java]
({{hadoop-hdds/container-service}}) builds real data and metadata volume sets
with {{failed.data.volumes.tolerated=1}} and a 5 second disk check timeout, and
uses the existing {{setDelegateChecker}} hook as the disk probe. The first
cycle fails the data volume; in the next {{checkAllVolumeSets}} cycle the
metadata volume is probed only after about 5000 ms, and {{checkAllVolumes}} on
an empty list takes about 5000 ms. A passing test means the defect is present.
h3. Patch
[^CR-2-return-at-once-for-empty-volume-list.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
{{checkAllVolumes}} returns an empty set at once for an empty collection, which
is what the timed out wait returns today. Non-empty collections are not
affected. Covered by {{testCheckAllVolumesWithNoVolumes}} in the existing
{{TestStorageVolumeChecker}}, which uses a 10 minute timeout and requires the
call to return within 10 seconds with no failed volumes; without the change it
times out in {{CountDownLatch.await}}. With it the volume checker and volume
set suites (73 tests) pass and checkstyle is clean. With the patch the
reproduction fails because the metadata volume is probed at once.
Found by code review of the datanode volume and per-disk RocksDB lifecycle
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
