caozhiqiang created HDFS-17936:
----------------------------------
Summary: Detect slow disks separately by storage type
Key: HDFS-17936
URL: https://issues.apache.org/jira/browse/HDFS-17936
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.6.0
Reporter: caozhiqiang
Assignee: caozhiqiang
The current slow disk detection does not differentiate between storage types.
When a DataNode has 1 SSD and 10 HDDs with
dfs.datanode.max.slowdisks.to.exclude=2 and
dfs.datanode.min.outlier.detection.disks=5, if the SSD is under heavy
read/write load, the single SSD may be identified as a slow disk. Client tasks
writing SSD data to that DataNode will then fail to select a volume and report:
No more available volumes.
{code:java}
public V chooseVolume(List<V> volumes, long replicaSize, String storageId)
throws IOException {
if (volumes.size() < 1) {
throw new DiskOutOfSpaceException("No more available volumes");
}
...
} {code}
Therefore, slow disk detection should be performed separately for each storage
type. If the number of disks of a given storage type is less than
dfs.datanode.min.outlier.detection.disks, detection for that storage type is
skipped entirely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
