Siyao Meng created HDDS-17261:
---------------------------------
Summary: Count disk I/O errors from the datanode data path per
volume
Key: HDDS-17261
URL: https://issues.apache.org/jira/browse/HDDS-17261
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode
Reporter: Siyao Meng
h3. Problem
When a read or write of container data or metadata hits an I/O error, the
datanode only schedules an on-demand volume check. The check writes a small
file, syncs it and reads it back. The file lands on new sectors and the read is
likely served from page cache, so the check keeps passing on a disk that fails
reads of existing data. Today such a disk only shows up as containers being
marked UNHEALTHY one at a time, and nothing reports that the errors come from
the same volume.
HDDS-11943 proposed failing the volume after enough of these errors and was
closed in favour of the degraded volume design in [PR
#8405|https://github.com/apache/ozone/pull/8405]. Its "IO Error Count" check
needs these errors to be counted per volume first.
h3. Proposal
Count disk I/O errors per data volume and report them, without changing the
volume state:
* Add {{DiskCheckUtil#isDiskIOError}}. It matches EIO, EROFS, ENXIO/ENODEV,
EUCLEAN and EREMOTEIO messages, and {{SyncFailedException}}, anywhere in the
cause chain. Missing files, ENOSPC, EMFILE and permission errors are not
counted.
* Add {{StorageVolumeUtil#onFailure(StorageVolume, Throwable)}}. Pass the
exception from the existing data path call sites: ChunkUtils read/write,
FilePerBlockStrategy, StreamDataChannelBase, KeyValueContainer, BlockUtils and
HddsVolumeUtil.
* Record disk I/O errors found by the container scanners as well.
* Add a {{NumDataPathIOErrors}} counter to {{VolumeInfoMetrics}}, plus a WARN
log naming the volume, logged at most once per
{{hdds.datanode.disk.check.min.gap}}.
A later change can feed this count into the degraded sliding window from the
design.
h3. Testing
Unit tests for the error classification, the per-volume count through
{{StorageVolumeUtil#onFailure}}, and recording from the container data scanner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
