[
https://issues.apache.org/jira/browse/HDFS-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated HDFS-17295:
-----------------------------
Description:
I encountered an error in one of our production environments.
Client error log is:
!image-2023-12-18-13-45-56-824.png|width=978,height=211!
namenode error log is:
!image-2023-12-18-13-58-25-102.png|width=974,height=373!
datanode capacity usage is:
!image-2023-12-18-14-07-05-802.png!
12 datanodes are all excluded because 7 is full and 5 is busy. 7 full is
obviously from datanode capacity usage. 5 busy can be derived from following
code:
!image-2023-12-18-14-25-12-447.png!
*considerLoadFactor* is set to 2 by default(controlled by
dfs.namenode.replication.considerLoad.factor)
*stats. getInServiceXceiverAverage()* is the total number of Xceivers divided
by the current number of datanodes in service.
In the error scenario mentioned above, the Xceiver count of 12 datanodes are:
0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24. Then the maxLoad is 2*(120/12)=20. The
last 5 datanodes will be excluded because 24 greater than 20.
Under the current settings, as long as more than half of the datanodes are
unavailable, the remaining available datanodes may be excluded due to high load.
More than half of datanodes are unavailable is not a rare scenario. Capacity
used up is one example. Storage policy is another example, suppose we has a 5
datanodes's cluster, 3 datanodes are all SSD, 2 datanodes are all HDD. The
storage policy for the /test/read and /test/write directories is HOT. Starting
from a certain moment, we read files in the/test/read directory, which lead 2
HDD datanodes's Xceiver high, then we try to put files into /test/write
directory, the put operation will fail and throw the similar exception
mentioned before.
was:
I encountered an error in one of our production environments.
Client error log is:
!image-2023-12-18-13-45-56-824.png|width=978,height=211!
namenode error log is:
!image-2023-12-18-13-58-25-102.png|width=974,height=373!
datanode capacity usage is:
!image-2023-12-18-14-07-05-802.png!
12 datanodes are all excluded because 7 is full and 5 is busy. 7 full is
obviously from datanode capacity usage. 5 busy can be derived from following
code:
!image-2023-12-18-14-25-12-447.png!
*considerLoadFactor* is set to 2 by default(controlled by
dfs.namenode.replication.considerLoad.factor)
*stats. getInServiceXceiverAverage()* is the total number of Xceivers divided
by the current number of datanodes in service.
In the error scenario mentioned above, the Xceiver count of 12 datanodes are:
0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24. Then the maxLoad is 2*(120/12)=20. The
last 5 datanodes will be excluded because 24 greater than 20.
Under the current settings, as long as more than half of the datanodes are
unavailable, the remaining available datanodes may be excluded due to high load.
More than half of datanodes are unavailable, which is not a rare scenario.
Capacity used up is one example. Storage policies is another example, suppose
we has a 5 datanodes's cluster, 3 datanodes are all SSD, 2 datanodes are all
HDD. The storage policy for the /test/read and/test/write directories is hot
> 'hdfs dfs -put' may fail when more than half of the datanodes are unavailable
> -----------------------------------------------------------------------------
>
> Key: HDFS-17295
> URL: https://issues.apache.org/jira/browse/HDFS-17295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.10.1
> Reporter: Xuze Yang
> Priority: Major
> Attachments: image-2023-12-18-13-45-56-824.png,
> image-2023-12-18-13-58-25-102.png, image-2023-12-18-14-07-05-802.png,
> image-2023-12-18-14-25-12-447.png
>
>
> I encountered an error in one of our production environments.
> Client error log is:
> !image-2023-12-18-13-45-56-824.png|width=978,height=211!
> namenode error log is:
> !image-2023-12-18-13-58-25-102.png|width=974,height=373!
> datanode capacity usage is:
> !image-2023-12-18-14-07-05-802.png!
> 12 datanodes are all excluded because 7 is full and 5 is busy. 7 full is
> obviously from datanode capacity usage. 5 busy can be derived from following
> code:
> !image-2023-12-18-14-25-12-447.png!
> *considerLoadFactor* is set to 2 by default(controlled by
> dfs.namenode.replication.considerLoad.factor)
> *stats. getInServiceXceiverAverage()* is the total number of Xceivers divided
> by the current number of datanodes in service.
> In the error scenario mentioned above, the Xceiver count of 12 datanodes are:
> 0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24. Then the maxLoad is 2*(120/12)=20.
> The last 5 datanodes will be excluded because 24 greater than 20.
> Under the current settings, as long as more than half of the datanodes are
> unavailable, the remaining available datanodes may be excluded due to high
> load.
> More than half of datanodes are unavailable is not a rare scenario. Capacity
> used up is one example. Storage policy is another example, suppose we has a 5
> datanodes's cluster, 3 datanodes are all SSD, 2 datanodes are all HDD. The
> storage policy for the /test/read and /test/write directories is HOT.
> Starting from a certain moment, we read files in the/test/read directory,
> which lead 2 HDD datanodes's Xceiver high, then we try to put files into
> /test/write directory, the put operation will fail and throw the similar
> exception mentioned before.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
