lfxy opened a new pull request, #8558:
URL: https://github.com/apache/hadoop/pull/8558
The current slow disk detection does not differentiate between storage
types. When a DataNode has 1 SSD and 10 HDDs with
dfs.datanode.max.slowdisks.to.exclude=2 and
dfs.datanode.min.outlier.detection.disks=5, if the SSD is under heavy
read/write load, the single SSD may be identified as a slow disk. Client tasks
writing SSD data to that DataNode will then fail to select a volume and report:
No more available volumes.
`public V chooseVolume(List<V> volumes, long replicaSize, String storageId)
throws IOException {
if (volumes.size() < 1) {
throw new DiskOutOfSpaceException("No more available volumes");
}
...
} `
Therefore, slow disk detection should be performed separately for each
storage type. If the number of disks of a given storage type is less than
dfs.datanode.min.outlier.detection.disks, detection for that storage type is
skipped entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
