[
https://issues.apache.org/jira/browse/HDFS-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14383:
-----------------------------------
Fix Version/s: 3.3.1
> Compute datanode load based on StoragePolicy
> --------------------------------------------
>
> Key: HDFS-14383
> URL: https://issues.apache.org/jira/browse/HDFS-14383
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
> Affects Versions: 2.7.3, 3.1.2
> Reporter: Karthik Palanisamy
> Assignee: Ayush Saxena
> Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-14383-01.patch, HDFS-14383-02.patch
>
>
> Datanode load check logic needs to be changed because existing computation
> will not consider StoragePolicy.
> DatanodeManager#getInServiceXceiverAverage
> {code}
> public double getInServiceXceiverAverage() {
> double avgLoad = 0;
> final int nodes = getNumDatanodesInService();
> if (nodes != 0) {
> final int xceivers = heartbeatManager
> .getInServiceXceiverCount();
> avgLoad = (double)xceivers/nodes;
> }
> return avgLoad;
> }
> {code}
>
> For example: with 10 nodes (HOT), average 50 xceivers and 90 nodes (COLD)
> with average 10 xceivers the calculated threshold by the NN is 28 (((500 +
> 900)/100)*2), which means those 10 nodes (the whole HOT tier) becomes
> unavailable when the COLD tier nodes are barely in use. Turning this check
> off helps to mitigate this issue, however the
> dfs.namenode.replication.considerLoad helps to "balance" the load of the DNs,
> upon turning it off can lead to situations where specific DNs are
> "overloaded".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
