[
https://issues.apache.org/jira/browse/HDFS-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-17538:
------------------------------
Description:
When decommissioning datanode, blocks will be checked one by one disk, then
blocks will be sent to trigger tranfer works in DN. This will make one disk of
decommissioning dn very busy and cpus stuck in io-wait with high loads, and
sometime even lead to OOM as below:
!image-2024-05-29-16-24-45-601.png|width=909,height=170!
!image-2024-05-29-16-26-58-359.png|width=909,height=228!
!image-2024-05-29-16-27-35-886.png|width=930,height=218!
Proposal to add priority queue for transfering blocks when decommisioning
datanode.
was:
When decommissioning datanode, blocks will be checked one by one disk, then
blocks will be sent to trigger tranfer works in DN. This will make one disk of
decommissioning dn very busy and cpus stuck in io-wait with high loads, and
sometime even lead to OOM as below:
!image-2024-05-29-16-24-45-601.png!
!image-2024-05-29-16-26-58-359.png!
!image-2024-05-29-16-27-35-886.png!
Proposal to add priority queue for transfering blocks when decommisioning
datanode.
> Add tranfer priority queue for decommissioning datanode
> -------------------------------------------------------
>
> Key: HDFS-17538
> URL: https://issues.apache.org/jira/browse/HDFS-17538
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Yuanbo Liu
> Priority: Major
> Attachments: image-2024-05-29-16-24-45-601.png,
> image-2024-05-29-16-26-58-359.png, image-2024-05-29-16-27-35-886.png
>
>
> When decommissioning datanode, blocks will be checked one by one disk, then
> blocks will be sent to trigger tranfer works in DN. This will make one disk
> of decommissioning dn very busy and cpus stuck in io-wait with high loads,
> and sometime even lead to OOM as below:
> !image-2024-05-29-16-24-45-601.png|width=909,height=170!
> !image-2024-05-29-16-26-58-359.png|width=909,height=228!
> !image-2024-05-29-16-27-35-886.png|width=930,height=218!
> Proposal to add priority queue for transfering blocks when decommisioning
> datanode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
