[
https://issues.apache.org/jira/browse/HDDS-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17261:
------------------------------
Status: Patch Available (was: Open)
> Count disk I/O errors from the datanode data path per volume
> ------------------------------------------------------------
>
> Key: HDDS-17261
> URL: https://issues.apache.org/jira/browse/HDDS-17261
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-17261.001.patch
>
>
> h3. Problem
> When a read or write of container data or metadata hits an I/O error, the
> datanode only schedules an on-demand volume check. The check writes a small
> file, syncs it and reads it back. The file lands on new sectors and the read
> is likely served from page cache, so the check keeps passing on a disk that
> fails reads of existing data. Today such a disk only shows up as containers
> being marked UNHEALTHY one at a time, and nothing reports that the errors
> come from the same volume.
> HDDS-11943 proposed failing the volume after enough of these errors and was
> closed in favour of the degraded volume design in [PR
> #8405|https://github.com/apache/ozone/pull/8405]. Its "IO Error Count" check
> needs these errors to be counted per volume first.
> h3. Proposal
> Count disk I/O errors per data volume and report them, without changing the
> volume state:
> * Add {{DiskCheckUtil#isDiskIOError}}. It matches EIO, EROFS, ENXIO/ENODEV,
> EUCLEAN and EREMOTEIO messages, and {{SyncFailedException}}, anywhere in the
> cause chain. Missing files, ENOSPC, EMFILE and permission errors are not
> counted.
> * Add {{StorageVolumeUtil#onFailure(StorageVolume, Throwable)}}. Pass the
> exception from the existing data path call sites: ChunkUtils read/write,
> FilePerBlockStrategy, StreamDataChannelBase, KeyValueContainer, BlockUtils
> and HddsVolumeUtil.
> * Record disk I/O errors found by the container scanners as well.
> * Add a {{NumDataPathIOErrors}} counter to {{VolumeInfoMetrics}}, plus a WARN
> log naming the volume, logged at most once per
> {{hdds.datanode.disk.check.min.gap}}.
> A later change can feed this count into the degraded sliding window from the
> design.
> h3. Testing
> Unit tests for the error classification, the per-volume count through
> {{StorageVolumeUtil#onFailure}}, and recording from the container data
> scanner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
