ZhenyuLi created HDFS-18026:
-------------------------------

             Summary: Stop excluding slow disks after 
dfs.datanode.max.slowdisks.to.exclude is reconfigured to 0
                 Key: HDFS-18026
                 URL: https://issues.apache.org/jira/browse/HDFS-18026
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 3.4.0
            Reporter: ZhenyuLi


h3. Problem

{\{dfs.datanode.max.slowdisks.to.exclude}} (added in HDFS-16371) is 
reconfigurable at runtime, and hdfs-default.xml documents that the value 0 
"disables excluding slow disk when choosing volume". However, after 
reconfiguring it from a positive value to 0, the DataNode keeps excluding the 
slow disks that were published before the change, until the DataNode is 
restarted.

h3. Root cause

{\{DataNodeDiskMetrics}} keeps the limit (\{{maxSlowDisksToExclude}}) and the 
published exclusion list (\{{slowDisksToExclude}}) as two separate fields:
* \{{setMaxSlowDisksToExclude()}}, called from 
\{{DataNode#reconfSlowDiskParameters}}, only updates the limit; it does not 
touch the list.
* The detector thread only assigns \{{slowDisksToExclude}} inside \{{if 
(maxSlowDisksToExclude > 0)}}, so while the limit is 0 the old list is never 
cleared.
* \{{FsVolumeList#chooseVolume}} filters candidate volumes by 
\{{getSlowDisksToExclude()}} without checking the limit.

h3. Impact

* Setting the value to 0 has no effect at runtime; previously excluded disks 
keep being skipped for new blocks.
* If the stale list covers every volume of the requested storage type (e.g. the 
only SSD volume of a DataNode, written with the ALL_SSD policy), 
\{{RoundRobinVolumeChoosingPolicy}} throws \{{DiskOutOfSpaceException("No more 
available volumes")}}, so that DataNode rejects writes for that storage type 
even though exclusion is supposed to be disabled.

h3. Steps to reproduce

# Enable disk metrics (\{{dfs.datanode.fileio.profiling.sampling.percentage}} > 
0) and set \{{dfs.datanode.max.slowdisks.to.exclude}} to 1.
# Wait until a slow disk is detected and 
\{{DataNodeDiskMetrics#getSlowDisksToExclude()}} is non-empty.
# Change the value to 0 and run \{{hdfs dfsadmin -reconfig datanode 
<host:ipc_port> start}}.
# \{{getSlowDisksToExclude()}} still returns the old disk, and new blocks are 
never placed on it.

h3. Proposed fix

Make \{{getSlowDisksToExclude()}} return an empty list whenever the limit is 
not positive, so disabling takes effect immediately and does not race with the 
detector thread. Also clear the list in the detector thread in that case, and 
make the field \{{volatile}} since it is written and read by different threads. 
A unit test in \{{TestFsVolumeList}} covers the reconfiguration.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to