[
https://issues.apache.org/jira/browse/HDDS-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned HDDS-17010:
---------------------------------
Assignee: Andy Huang
> Datanode volume check waits the full disk check timeout on an empty volume
> set, delaying the check of the next volume set
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17010
> URL: https://issues.apache.org/jira/browse/HDDS-17010
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Andy Huang
> Priority: Minor
> Attachments: CR-2-return-at-once-for-empty-volume-list.patch,
> TestBugCR2EmptyVolumeSetCheck.java
>
>
> h3. Mechanism
> {{StorageVolumeChecker.checkAllVolumes}} creates a {{CountDownLatch}} of 1
> and counts it down only inside the loop over the volumes (in the result
> callback, or when the last volume's check is skipped). For an empty
> collection the loop does not run, so {{latch.await}} always waits the full
> {{hdds.datanode.disk.check.timeout}} (10 minutes by default) and then logs
> "checkAllVolumes timed out". {{checkAllVolumeSets}} is synchronized and runs
> the sets one after the other in registration order (data, metadata, db) on
> one thread, so every set after the empty one is checked that much later, in
> every periodic cycle. The periodic check is scheduled with a fixed delay, so
> every later cycle also starts that much later, even when the empty set is the
> last one.
> h3. Trigger
> # {{hdds.datanode.failed.data.volumes.tolerated}} (or the metadata or db
> variant) is set to at least the number of volumes in that set. The tolerance
> is not capped at the number of configured volumes.
> # Every volume in that set fails a check. {{hasEnoughVolumes}} is still true,
> so the datanode keeps running with the empty set registered.
> # Each later periodic check waits the timeout on the empty set.
> An empty set cannot occur at startup ({{initializeVolumeSet}} fails) or with
> the default tolerance of -1 (the datanode stops when the set becomes empty).
> h3. Impact
> * Failure detection for the volume sets registered after the empty one is
> delayed by the disk check timeout in every cycle, and every later cycle
> starts that much later. For metadata and db volumes the periodic check is the
> only check (argued from the code).
> * A misleading "checkAllVolumes timed out" warning is logged every cycle.
> * No data impact.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugCR2EmptyVolumeSetCheck.java]
> ({{hadoop-hdds/container-service}}) builds real data and metadata volume sets
> with {{failed.data.volumes.tolerated=1}} and a 5 second disk check timeout,
> and uses the existing {{setDelegateChecker}} hook as the disk probe. The
> first cycle fails the data volume; in the next {{checkAllVolumeSets}} cycle
> the metadata volume is probed only after about 5000 ms, and
> {{checkAllVolumes}} on an empty list takes about 5000 ms. A passing test
> means the defect is present.
> h3. Patch
> [^CR-2-return-at-once-for-empty-volume-list.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> {{checkAllVolumes}} returns an empty set at once for an empty collection,
> which is what the timed out wait returns today. Non-empty collections are not
> affected. Covered by {{testCheckAllVolumesWithNoVolumes}} in the existing
> {{TestStorageVolumeChecker}}, which uses a 10 minute timeout and requires the
> call to return within 10 seconds with no failed volumes; without the change
> it times out in {{CountDownLatch.await}}. With it the volume checker and
> volume set suites (73 tests) pass and checkstyle is clean. With the patch the
> reproduction fails because the metadata volume is probed at once.
> Found by code review of the datanode volume and per-disk RocksDB lifecycle
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
