Mingliang Liu created HDFS-10335:
------------------------------------
Summary: Mover$Processor#chooseTarget() always chooses the first
matching target datanode
Key: HDFS-10335
URL: https://issues.apache.org/jira/browse/HDFS-10335
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer & mover
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Critical
Currently the
{{org.apache.hadoop.hdfs.server.mover.Mover$Processor#chooseTarget()}} always
chooses the first matching target datanode from the candidate list. This may
make the mover schedule a lot of task to a few of the datanodes (first several
datanodes of the candidate list). The overall performance will suffer
significantly from this because of the saturated network/disk usage. Specially,
if the {{dfs.datanode.balance.max.concurrent.moves}} is set, the scheduled move
task will be queued on a few of the datanodes, regardless of other available
storage resources. We need an algorithm which can distribute the move tasks
approximately even across all the candidate target datanodes (storages).
Thanks [~szetszwo] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
