[
https://issues.apache.org/jira/browse/HDFS-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798040#comment-17798040
]
Xuze Yang commented on HDFS-17295:
----------------------------------
A simple solution is to increase the considerLoadFactor. For example, if we set
considerLoadFactor to 4, then more than three-quarters of datanodes are
unavailable may lead to put operation fail. More generally, if
considerLoadFactor is N, then only more than (N-1)/N datanodes are unavailable
may lead to put operation fail.
However, in my opinion, simply increasing the considerLoadFactor is not good
enough. Because a larger considerLoadFactor value means a more uneven load,
which may lead to a decrease in the read and write performance of the datanode.
> 'hdfs dfs -put' may fail when more than half of the datanodes are unavailable
> -----------------------------------------------------------------------------
>
> Key: HDFS-17295
> URL: https://issues.apache.org/jira/browse/HDFS-17295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.10.1
> Reporter: Xuze Yang
> Priority: Major
> Attachments: image-2023-12-18-13-45-56-824.png,
> image-2023-12-18-13-58-25-102.png, image-2023-12-18-14-07-05-802.png,
> image-2023-12-18-14-25-12-447.png
>
>
> I encountered an error in one of our production environments.
> Client error log is:
> !image-2023-12-18-13-45-56-824.png|width=978,height=211!
> namenode error log is:
> !image-2023-12-18-13-58-25-102.png|width=974,height=373!
> datanode capacity usage is:
> !image-2023-12-18-14-07-05-802.png!
> 12 datanodes are all excluded because 7 is full and 5 is busy. 7 full is
> obviously from datanode capacity usage. 5 busy can be derived from following
> code:
> !image-2023-12-18-14-25-12-447.png!
> *considerLoadFactor* is set to 2 by default(controlled by
> dfs.namenode.replication.considerLoad.factor)
> *stats. getInServiceXceiverAverage()* is the total number of Xceivers divided
> by the current number of datanodes in service.
> In the error scenario mentioned above, the Xceiver count of 12 datanodes are:
> 0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24. Then the maxLoad is 2*(120/12)=20.
> The last 5 datanodes will be excluded because 24 greater than 20.
> Under the current settings, as long as more than half of the datanodes are
> unavailable, the remaining available datanodes may be excluded due to high
> load.
> More than half of datanodes are unavailable is not a rare scenario. Capacity
> used up is one example. Storage policy is another example, suppose we has a 5
> datanodes's cluster, 3 datanodes are all SSD, 2 datanodes are all HDD. The
> storage policy for the /test/read and /test/write directories is HOT.
> Starting from a certain moment, we read files in the/test/read directory,
> which lead 2 HDD datanodes's Xceiver high, then we try to put files into
> /test/write directory, the put operation will fail and throw the similar
> exception mentioned before.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
