[
https://issues.apache.org/jira/browse/HDFS-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated HDFS-17295:
-----------------------------
Description:
I encountered an error in one of our production environments.
Client error log is:
!image-2023-12-18-13-45-56-824.png|width=978,height=211!
namenode error log is:
!image-2023-12-18-13-58-25-102.png|width=974,height=373!
datanode capacity usage is:
!image-2023-12-18-14-07-05-802.png!
12 datanodes are all excluded because 7 is full and 5 is busy. 7 full is
obviously from datanode capacity usage. 5 busy can be derived from following
code:
!image-2023-12-18-14-25-12-447.png!
*considerLoadFactor* is set to 2 by default(controlled by
dfs.namenode.replication.considerLoad.factor)
*stats. getInServiceXceiverAverage()* is the total number of Xceivers divided
by the current number of datanodes in service.
In the error scenario mentioned above, the Xceiver count of 12 datanodes are:
0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24. Then the maxLoad is 2*(120/12)=20. The
last 5 datanodes will be excluded because 24 greater than 20.
Under the current settings, as long as more than half of the datanodes are
unavailable, the remaining available datanodes may be excluded due to high load.
More than half of datanodes are unavailable, which is not a rare scenario.
Capacity used up is one example. Storage policies is another example, suppose
we has a 5 datanodes's cluster, 3 datanodes are all SSD, 2 datanodes are all
HDD. The storage policy for the /test/read and/test/write directories is hot
was:
I encountered an error in one of our production environments.
Client error log is:
!image-2023-12-18-13-45-56-824.png|width=978,height=211!
namenode error log is:
!image-2023-12-18-13-58-25-102.png|width=974,height=373!
datanode capacity usage is:
!image-2023-12-18-14-07-05-802.png!
12 datanodes are all excluded because 7 is full and 5 is busy.
> 'hdfs dfs -put' may fail when more than half of the datanodes are unavailable
> -----------------------------------------------------------------------------
>
> Key: HDFS-17295
> URL: https://issues.apache.org/jira/browse/HDFS-17295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.10.1
> Reporter: Xuze Yang
> Priority: Major
> Attachments: image-2023-12-18-13-45-56-824.png,
> image-2023-12-18-13-58-25-102.png, image-2023-12-18-14-07-05-802.png,
> image-2023-12-18-14-25-12-447.png
>
>
> I encountered an error in one of our production environments.
> Client error log is:
> !image-2023-12-18-13-45-56-824.png|width=978,height=211!
> namenode error log is:
> !image-2023-12-18-13-58-25-102.png|width=974,height=373!
> datanode capacity usage is:
> !image-2023-12-18-14-07-05-802.png!
> 12 datanodes are all excluded because 7 is full and 5 is busy. 7 full is
> obviously from datanode capacity usage. 5 busy can be derived from following
> code:
> !image-2023-12-18-14-25-12-447.png!
> *considerLoadFactor* is set to 2 by default(controlled by
> dfs.namenode.replication.considerLoad.factor)
> *stats. getInServiceXceiverAverage()* is the total number of Xceivers divided
> by the current number of datanodes in service.
> In the error scenario mentioned above, the Xceiver count of 12 datanodes are:
> 0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24. Then the maxLoad is 2*(120/12)=20.
> The last 5 datanodes will be excluded because 24 greater than 20.
> Under the current settings, as long as more than half of the datanodes are
> unavailable, the remaining available datanodes may be excluded due to high
> load.
> More than half of datanodes are unavailable, which is not a rare scenario.
> Capacity used up is one example. Storage policies is another example, suppose
> we has a 5 datanodes's cluster, 3 datanodes are all SSD, 2 datanodes are all
> HDD. The storage policy for the /test/read and/test/write directories is hot
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
